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EUCALYPTUS:  INTEGRATING  NATURAL  LANGUAGE  INPUT  WITH  A 

GRAPHICAL  USER  INTERFACE 


INTRODUCTION 

As  interactive  software  tools  of  the  future  become  mote  and  more  complex  and  knowledge-inten¬ 
sive,  the  more  the  so-called  “human-computer  dialogue”  may  come  to  resemble  true  human-human  dia¬ 
logue  in  a  number  of  ways.  Foremost  among  these  is  discourse  processing,  the  human  ability  to  track  and 
maintain  continuity  of  topic,  reference,  and  reasoning  in  extetided  sequences  of  natural  language  (NL) 
utterances,  whether  monologue  or  dialogue.  At  its  simplest  level,  discourse  processing  allows  speakers  or 
writers  to  use  a  number  of  convenient  shorthand  devices  to  eliminate  excessive  r^tition  and  wordiness, 
with  the  discourse  context  serving  to  fill  in  gaps  aiKl  resolve  underspecifies.  At  a  higher  level  of  complex¬ 
ity,  discourse  structure  reflects  the  logical  progtessim  of  thou^ts  in  a  discussion  and  signals  shifts  from 
one  topic  to  another,  while  at  the  highest  levels,  it  transitions  to  the  less  empirically  explicable  arts  of  rhe¬ 
torical  and  compositional  style. 

hi  the  early  days  of  artificial  intelligence  research,  many  believed  that  once  the  technology  had 
sufficiently  advanc^,  would  become  the  human-emnputer  interface  medium  of  choice.  The  success  of 
the  grai^cai  user  interface  (GUI)  in  the  intervening  years  now  suggests  that  each  of  these  interface  media 
has  relative  advantages  and  disadvantages  [1].  Most  GUIs  currently  in  use  are  based  on  the  highly  natural 
pariidigm  of  direct  manipulation,  in  which  the  user  operates  directly  on  visual  analogs  of  dmnain  and  con¬ 
trol  entities  (represented  as  menus,  dials,  iconic  objects,  etc.)  using  manual  pointing,  seizing,  and  sdection 
gestures.  Such  interfaces  can  become  tiring  and  repetitious  in  use,  however,  because  of  their  inability  to 
abstraa  over  previous  interactions  and  the  necessity  to  select  command  arguments  individually  rafiier  than 
denotatitmally.  The  NL  paradigm,  conversely,  is  ba^  on  the  mutual  reasoning  ability  and  underlying  con¬ 
textual  knowledge  of  ^  participating  patties,  so  it  provides  many  natural  medianisms  for  abstraction, 
sudi  as  logical  connectives,  quantification,  and  generalized  descriptiotL  But  by  the  same  token,  NL  runs 
the  risk  of  being  underspedfic,  sometimes  resulting  in  vagueness,  ambiguity,  and  misunderstanding. 

Since  discourse  understanding  depends  twavily  on  contextual  knowledge,  infermcing  ability,  and 
inqrlidt  understanding  of  the  principles  of  effective  conununication,  a  human-computer  interface  having 
th^  C2q>at»lities  falls  into  a  category  that  in  recoit  years  has  come  to  be  known  as  the  Intelligettt  User 
Interface,  or  lUI  [2].  The  role  of  the  lUI  is  to  serve  as  an  intelligent  intermediary  betweoi  user  and  applica- 
titni,  translating  cognitively  high-level  user  irqruts  into  lower-level  (^ratitms  that  the  af^lication  can  exe¬ 
cute,  and  custranizing  the  form  and  modality  of  syst^  ouqxit  for  maximum  user  comprehensidlity.  With 
the  addititHi  of  discursive  context  tracking,  each  hansaction  between  user  and  system  can  be  minimally 
qredfic,  deriving  its  full  interpretation  from  the  dialogue  partners’  implicit  understanding  of  the  curteitt 
topic  of  the  interchange.  A  recoit  trend  in  lUI  technology  is  toward  integrated  interfaces  in  >^ch  botfi  the 
graphical  and  NL  paradigms  play  complementary  roles,  die  strengths  of  the  one  supporting  the  weaknesses 
of  the  other  [3,4],  and  it  has  also  been  suggested  that  discourse  modeling  techniques  might  be  ^licable  to 
grrqjhical  interactitnis  as  well  [5]. 
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This  report  describes  a  natural  language  interface  with  limited  discourse  capability  that  the  author 
has  built  and  integrated  into  the  GUI  of  a  Navy  system  design  and  training  demonstratitm.  the  KOALAS 
(Knowledgeable,  Observation  Analysis-Linked  Advisory  System)  Test  banning  Tool,  as  part  of  an  explor¬ 
atory  development  effort  demonstrating  intelligent  multimedia  interface  technology  ai^lied  to  Navy  needs. 
The  interface,  named  Eucalyptus  (“natural  input  to  koalas”),  handles  both  imperative  commands  and  data¬ 
base  queries,  either  spoken  t^ugh  a  mictof^ne  or  typed  at  the  keyboard,  while  still  allowing  full  use  of 
the  original  GUI  as  an  alternative  input  medium.  The  NL  and  grafdiical  interfaces  do  not  sim[dy  coexist  but 
are  mutually  integrated,  interacting  in  a  number  of  ways  that  are  discussed  in  detail. 

Integrating  NL  input  into  an  existing  GUI  (rather  than  developing  a  fully  integrated  multimedia 
interface  from  scratch)  facilitated  the  development  of  a  demonstration  for  the  Navy  of  lUI  technology 
api^ed  to  one  of  their  own  tools,  and  also  provided  an  avenue  for  exploring  which  gr^rhical  interaction 
tet^ques  are  more  compatible  with  NL  arul  which  less.  Eucalyptus  also  builds  on  earlier  work  in  NL 
interfa^  by  our  group  [7],  giving  us  an  opportunity  to  further  exercise  the  natural  language  processing 
system  we  have  been  developing  over  the  past  several  years,  with  a  particular  view  toward  extending  its 
discourse  handling  capability. 

The  report  is  organized  as  follows.  It  begins  with  an  introduction  to  the  KOALAS  demonstration 
tool  and  its  grai^cal  user  interface.  This  is  followed  by  a  statement  of  the  basic  working  concerts  that 
were  adopted  for  diis  project.  The  report  then  describes  the  Eucalyptus  integrated  NL  interface,  beginning 
with  the  basic  NL  equivalents  for  each  graphical  interaction  technique,  followed  by  a  discussion  of  the  spe¬ 
cial  crqrabilities  that  NL  offers,  and  concluding  with  a  description  of  how  NL  and  gnq^cs  interact  in  the 
integrs^  interface.  It  then  describes  the  system  architecture  and  details  of  the  implementation,  presents 
some  confirmatory  user  data,  gives  an  overview  of  related  research,  and  concludes  with  a  discussion  of 
findings  and  observations. 

KOALAS  AEW  TEST  PLANNING  TOOL 

KOALAS  [8]  is  a  process  architecture  for  the  design  of  a  new  generation  of  intelligent  control  sys¬ 
tems  in  which  the  inductive  intelligence  of  a  human  operator  is  optimally  joined  with  the  deductive  c^- 
Inlities  of  machine  intelligence.  The  KOALAS  AEW  (Airborne  Early  Warning)  Test  banning  Tool  [9]  is 
an  implication  built  by  the  Los  Alamos  National  Laboratory  for  the  Naval  Air  Systems  Command  to  serve 
as  a  coneqx  danonstration  for  this  design  philosophy.  The  Test  Planning  Tool  illustrates  how  a  simulation- 
based  software  environment  built  around  the  KOALAS  concept  of  intelligent  control  can  be  used  in  the 
design  of  complex  military  systems,  in  particular  tlK  plarming  of  operational  and  design  tests;  the  tool’s 
authors  also  suggest  other  possiUe  implications,  notably  tactical  combat  training. 

The  particular  mOitary  system  this  tool  simulates  is  a  hypothetical  command  and  control  (C^)  sys- 
tm  for  a  Navy  E2  AEW  aircraft.  The  tool  consists  of  two  components,  a  scenario  goreration  program  and 
a  simulation  program,  of  which  we  are  only  concerned  with  the  second.  The  simidation  program  runs  on 
Sun  SPARCstations  and  has  a  griqMcal  interface  writtrai  using  die  Sun>^ew  toolkit.  Central  to  the  grafdii- 
cal  display  (Hg.  1)  is  a  syndietic  radar  screoi  showing  symbolic  blips  for  each  friendly  aircraft  and  actual 
or  suspect  incoming  raid  in  the  simulated  radar  field.  This  screen  is  accompanied  by  four  scrollable  text 
di^lay  windows  diat  are  continuously  updated  to  show  current  fighter  and  threat  status,  communications 
transactions,  and  sensor  readings.  The  system  also  contains  a  rule-based  tactical  advice  goierator  that  mon¬ 
itors  die  state  of  the  simulation  and  proposes  courses  of  action  in  die  form  of  simple  imperative  English 
soitences.  The  operator  can  either  let  the  simulation  run  autonomously  by  putting  the  advice  goierator  in 
“auto  accept"  mode  (causing  the  system  to  automatically  execute  each  internal  tactical  proposal),  or  inter- 


*An  overview  of  an  earlier  venkm  of  the  interface  can  be  found  in  Ref.  6. 
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Fig.  1  -  KOALAS  grqjhical  user  interface 


vene  in  the  simulation  at  any  time  by  accepting  individual  pieces  of  advice,  manually  issuing  orders  to 
fighters,  or  inserting  hypotheses  about  the  location  and  identity  of  possible  incoming  raids. 

Operator  intervention  takes  place  through  a  number  of  different  graphical  interaction  techniques. 
The  operator  selects,  views,  and  accepts  system  advice  using  buttons  in  an  Advice  Window,  and  generates 
fighter  orders  and  threat  hypotheses  by  selecting  the  desired  command  from  a  menu  and  then  entering  data 
into  a  popup  dialogue  window  that  appears  on  the  screen.  This  window  prompts  the  operator  to  use  the 
mouse  to  left-click  on  entities  in  the  radar  display,  each  click  automatically  updating  the  ^propriate  selec¬ 
tion  items  and  text  fields  in  the  window.  A  ri^t  click  undoes  the  most  recent  data  entry,  clearing  the  corre¬ 
sponding  text  fields  and  backing  up  to  the  previous  prompt.  When  all  data  have  been  entered,  a  final  left 
click  anywhere  on  the  scteoi  (or  a  click  on  the  window’s  Accept  Data  button)  grants  final  accei^ce,  exe¬ 
cuting  the  command;  the  window  also  provides  a  Cancel  button  to  abort  the  proposed  operation. 

The  operator  can  also  modify  various  aspects  of  the  simulation  and  graphical  display.  A  middle 
mouse  click  on  fire  radar  screen  re-centers  it  at  those  coordinates.  Holding  the  left  mouse  buttrni  down  over 
an  aircraft  or  threat  icon  displays  a  popup  window  giving  such  pertinent  information  as  ID,  aircraft  type, 
status,  bearing,  course,  and  range,  and  a  right  click  on  a  UAV  (Unmanned  Airborne  Vehicle,  a  drone  sur¬ 
veillance  aircraft)  or  hypothetical  threat  displays  icons  representing  its  associated  threat  sightings.  Finally, 
an  Experimenter  Control  panel  provides  selection  items,  buttons,  and  check  boxes  for  such  operations  as 
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diangbig  the  simulaticMi  nintiine  speed,  zoraiing  the  radar  screen,  generating  visible  trails  bdiind  aircraft 
and  radar  Mips,  suq)ending  the  simulatitMi,  and  exiting  the  i»ogram. 

The  variety  of  gn^cal  interaction  techniques  -  buttons,  moius,  dialogue  windows,  (m/off 
switches,  choice  items,  text  fields,  and  mousing  cm  icons  or  screen  positions  -  suggested  that  this  interface 
migjitt  be  a  finuthil  testbed  for  exfrioiing  the  integiatimi  of  NL  and  graidiical  input  At  the  same  time,  how¬ 
ever.  it  was  evident  that  the  software  had  beat  designed  as  a  concept  demonstration  rather  than  a  {Hoduc- 
titm  qutpity  tool,  and  that  its  developers  were  well  aware  that  the  graphical  interface  they  had  provided  was 
cursory  at  best  describing  its  design  diemselves  as  '^superficial”  [9].  This  meant  that  care  would  have  to  be 
taken  not  to  misuiteriHet  specific  design  flaws  in  the  KOALAS  irtterface  as  indicative  of  graj^cal  inter¬ 
faces  in  general,  while  ^  correctly  identifying  those  that  do  tend  to  exist  in  one  form  or  another  even  in 
the  majority  of  currait  state-of-the-art  windowing  systems. 

The  tool’s  GUI  also  proved  interesting  frmn  an  NL  standpoint  in  another  way:  the  grai^cal  inter¬ 
action  technique  used  for  filter  management  and  threat  assessment  bears  an  intriguing  resmUance  to  NL 
syntax  and  reference.  The  item  labels  on  the  command  selection  menus  are  all  simple  verbs,  and  in  most 
cases  the  fields  of  die  dialogue  windows  correspond  to  the  verb’s  linguistic  argumoits  (verb  complements). 
For  examffle,  to  order  a  fighter  to  a  new  assignmem  station,  the  curator  first  selects  the  Move  item  from 
die  Halter  Management  menu,  dien  elides  on  the  fighter,  and  then  on  the  destination  -  a  sequence  of  t^r- 
atimis  closely  resemlding  the  imperative  sentoice  Move  this  fighter  here.  This  use  of  the  mouse  to  denote 
or  ictentify  an  object  in  a  command  omtext  is  not  direct  manipulation  as  such  but  more  pn^riy  direa 
gnqihical  reference,  and  referoice  is  a  key  ingredient  of  NL  or  NL-like  interactions.  A  simulation-based 
system  like  this  (Hie,  embodying  as  it  does  a  domain-constrained  world  of  dynamic  entities  engaged  in  var¬ 
ious  acticHis  and  possessing  various  attributes,  provides  a  good  arena  for  investigating  refermice-related 
issues. 


This  work  is  not  intended  to  suggest  that  an  actual  military  system  such  as  the  (me  the 
KOALAS  tool  simulates  is  an  ^ropriate  target  for  spokmi  NL  irqmt,  particularly  when  new  and  e^qieri- 
mental  techniques  like  discourse  tracking  are  being  investigated.  The  problmns  of  speaker  independence, 
qieech  recogniti(Hi  accuracy,  habitable  coverage,  and  NL  ambiguity  have  not  yet  b^  solved  s^cientiy 
to  make  truly  flexiUe  NL  interfaces  to  mission-critical  military  systems  feasible  at  diis  time.  Since  the 
KOALAS  tool  is  not  a  prototype  of  su(b  a  system  but  rather  an  interactive  tool  tiiat  illustrates  how  su(b 
prcHotypes  might  be  designed  mA  tested,  it  closely  resembles  a  decision  aid  or  decision  support  syston:  the 
user  makes  det^sirms  Imsed  (m  informatitm  obtairied  ftom  die  system  and  itiftirms  die  system  of  those  deci¬ 
sions  so  it  can  tqpdate  its  internal  situational  representadtm.  As  such  ncmcridcal  desktop  systems  beermae 
more  and  more  knowledge-intensive,  the  more  useful  a  flexible  and  intelligent  interface  with  spokm  NL 
irqwt  ciqiatdlity  like  Eucalyptus  can  be. 

WORKING  ASSUMPTIONS 

The  devdtqHiient  and  (tesign  of  Eucalyptus  was  grounded  in  a  number  of  initial  worldrtg  assump- 
ti(His  aai  decisiocis. 

Cominercial  Spcedi  Recognition.  Eucalyptus  uses  a  self-contained,  ctHnmercial  qpeetdi  recogni¬ 
tion  ftom  end  and  so  does  not  involve  any  reseandi  in  the  speetb  recognititm  process  itself. 

Natural  Language  Only.  The  Eucalyptus  user  is  to  interact  with  die  syston  using  nonndive 
EngUdi  idtaances  only.  These  ctm  inclixte  dliptical  and  flragmoitary  forms,  but  (Hily  those  diat  pet^e 
might  noimdly  use  wiftesKdiodier  under  similar  (dreumstimees.  Eucalyptus  is  dnis  awoken  NL  irqmt  sys¬ 
tem  radier  dian  simffly  a  voice  input  system.  This  decision  arises  from  our  groiqi’s  history  of  taking  a 
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stnmgly  linguistic  approach  to  NL  processing,  one  of  our  cunent  research  goals  being  to  study  how 
human-human  discourse  and  dialogue  pix^rties  might  be  carried  over  to  human-computer  ittteractions. 
Our  interests  are  dius  not  with  interface  modality  or  human-computer  interaction  techniques  as  sudi,  but 
with  the  role  natural  language  and  discourse  processing  might  have  in  more  intelligent  interfaces  of  die 
future. 


Input  Only.  Aside  from  generating  brief  Imt  helpful  English  sentence  fragments  in  respmise  to 
user  queries.  Eucalyptus  ignores  NL  ouqwt  issues  such  as  generatitm  or  speech  synthesis.  The  addition  of 
spt^en  NL  output  to  the  KOALAS  interface  and  the  Eucalyptus  query  facility  is  currently  under  investiga¬ 
tion  by  other  researchers  in  the  Interactive  Systems  group  at  the  Nav^  Research  Laboratoiy  (NRL). 

Graphical  Interface  Unchanged.  In  Eucalyptus  the  tool's  original  interface  is  left  undianged. 
The  objective  was  not  to  improve  the  interaction  techniques  and  displays  of  the  GUI  or  to  extend  its  func¬ 
tionality  in  any  way,  but  simply  to  integrate  NL  whenever  possible  with  those  grajMcal  constructs  that 
were  dready  provided  and  learn  from  the  experience. 

No  New  Domain  Functionality.  Eucalyptus  does  not  give  the  NL  interface  any  special  domain- 
specific  functionality  not  already  availaUe  in  some  way  via  the  graphical  interface.  This  approach  assumes 
that  the  existing  GUI  represents  a  model  of  all  the  operational  functionality  die  syston's  designers  intended 
for  it  to  {xovide  and  support,  and  that  the  NL  interftux  should  adhere  to  the  same  (implicit)  VO  specifica¬ 
tion.  For  example,  although  an  NL  interface  to  the  tactical  advice  generator  (i.e.,  a  query-based  explanation 
facility)  might  be  a  desirable  addition  to  the  syston,  it  would  require  extending  the  underlying  artificial 
intelligence  capabilities  of  the  KOALAS  triplication  itself,  not  just  making  the  NL  interface  more  intdli- 
gem. 


Datalink,  not  Voice  Comms.  Since  die  KOALAS  tool  simulates  a  command  and  control  statitm 
that  would  replace  the  E2’s  current  radar  screen  and  li^t-pen  interface  to  die  E2-F14  datalink,  one  might 
expect  the  role  of  NL  input  to  be  simulation  of  the  voice  communications  channel  between  E2  controller 
and  fighter  pilot  Since  Eucalyptus  treats  the  tool  as  a  decision  support  system  rather  than  as  a  prototype 
system,  however,  it  uses  NL  to  interact  not  with  die  simulated  aircraft  (using  standard  Qmim  Brevity 
vocabulary  and  syntax)  but  with  the  computer  systm  itself  O-c..  the  datalink)  just  as  the  graphical  interface 
does. 


User-Initiated  NL  Interactions  Only.  The  original  KOALAS  grtqMcal  interface  goierates  diree 
types  of  NL-like  textual  omputs:  fighter  status  entries  (“Vector  track  I”),  tactical  advice  C‘Deidoy  fighter 
18  to  station  3“),  and  advice  explanations  (fairly  len^y  English-like  rule  traces).  These  are  formatted 
ASCn  strings  goierated  by  template  filling,  and  not  having  NL  semantic  representations,  Eucalyptus  does 
not  consider  diem  part  of  ^  NL  dialogue  and  so  does  not  interact  with  them  as  such.  Similarly,  events  ini¬ 
tiated  by  die  systmn  (such  as  the  [miposal  of  a  new  hypothetical  threat  or  the  automatic  execution  of  tacti¬ 
cal  advice)  are  not  construed  as  implicit  “utterances”  to  be  included  in  the  NL  discourse  context,  aldiou^ 
treating  them  as  sudi  is  a  possibility  worth  investigating. 

No  Graphical  Metalanguage.  Eucalyptus  treats  the  grtqihical  and  NL  interfaces  as  parallel  and 
sonantically  equivalent,  eadi  accessing  the  same  set  of  domain-functional  operatitHis  but  using  different 
lexicaVsyntactic  medianisms  to  do  so.  Fbr  example,  a  GUI  “sentence"  to  initiate  aircraft  trail  display  might 
consist  of  two  “{Aliases”:  clicking  the  Experimenter  Control  button  to  bring  up  the  control  panel,  and 
then  clicking  the  panel’s  Aircraft  IWdls  switch.  The  NL  sentence  with  the  sanm  semantics  is  simply  Show 
tdrcrtft  trails,  not  “Open  the  Experimenter  Control  panel  and  push  the  Aircraft  Trails  switch.”  The  latter  is 
m  examine  of  metalanguage,  or  language  talking  about  language  -  in  this  case,  using  NL  to  refer  explicitly 
to  the  vocabulary  items  (control  panel,  switch)  and  syntax  (order  of  grtqdtical  operations)  of  die  gr^diical 
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language.  By  disaDowing  such  expressions,  di^t  manipulation  ii^xit  devices  are  honored  as  just 
that,  entities  to  be  manipulated  diiet^y  by  the  user,  not  indirectly  by  being  talked  about. 

Similarly,  a  domain  object  represented  by  an  ictm  on  the  KOALAS  radar  screen  can  be  referenced 
in  tmma  of  the  erttity  tire  icon  derxrtes  (e.g.,  Ms  fighter)  but  not  metalinguisticaUy  C'this  icem”).  Finally,  a 
Eucalyptus  command  or  query  that  results  in  ouqxit  to  the  graphical  diqday  must  be  i^uased  in  terms  of 
tire  diqdayed  enti^  rather  than  the  display  device  -  for  examine  Show  M  advice  explanation,  not  “Open 
the  Advice  Exfdanation  window.”  While  the  latter  is  something  a  computer  expert  might  be  inclined  to  say, 
the  average  user  would  jMObaUy  trot  resort  to  tiut  extra  level  of  indirection. 

The  role  of  tire  NL  interface  in  Eucalyptus  is  thus  not  to  verbally  operate  the  graphical  interface 
ctmtnds,  but  to  directly  interaa  with  the  underlying  api^catitm  program  just  as  the  graphical  interface 
does.  An  analogy  wot^  be  a  voice-activated  home  lighting  system  that  can  also  be  accessed  by  wall 
switches  bearing  labels  like  “Ftont  Door  Lights”.  One  can  either  flip  the  switch  bearing  that  label  or  say 
“Turn  on  the  fitmt  door  lights,”  which  is  a  reference  not  to  the  switdi  itself  (or  to  its  label)  but  to  the  same 
entity  controlled  by  the  switch  (and  denoted  by  its  label).  This  sqjproach  would  disaUow  the  utterance  “Flip 
the  fixmt  door  tight  switch”  since  the  switch  is  for  direct  manipulation  only:  we  don’t  need  the  physical 
interaction  device  to  change  state,  just  the  entity  it  crmtrols. 

NL  INTERFACE:  BASIC  GRAPHICAL  EQUIVALENTS 

As  described  earlier,  the  KOALAS  tool  grrqrhical  interface  provides  a  number  of  differeru  interac- 
tkm  tedhnityies  for  user  input:  menus,  dialogue  windows,  check  boxes,  cycle  items,  buttons,  and  mouse 
didcs  <xi  the  radar  screoi.  Each  poses  difierent  problems  fm^  the  design  of  equivalent  NL  utterances  that 
interact  qrpropriately  witii  the  grrqrhical  devices  in  the  integrated  interface.  We  will  look  at  each  of  tiiese 
techniques  in  turn,  beginning  with  the  menu/dialogue  window  cmnbination  used  for  fighter  management 
and  ttueat  assessment 

Fighter  Management 

Six  order  types  ate  available  from  the  Figluer  Management  meim  in  tire  GUI;  Move  (ord»  a 
dqtioyed  aircraft  to  a  new  assignment  statirm).  Deploy  (order  an  aircraft  fimn  tire  carrier  to  an  assignmoit 
station).  Vector  (order  a  fighter  out  to  a  hypothetical  threat’s  coordinates),  Reftid  (order  an  aircraft  to  a 
tanker  for  refueling).  Recall  (order  an  aircraft  back  to  the  carrier),  and  Relieve  (order  a  backup  fighter  to 
refieve  a  forward  fi^iter  for  tefoeling). 

Selecting  an  order  type  Ixings  up  a  dialogue  window  prompting  the  user  to  didc  on  one  or  mote 
objects  oa  the  radar  screoi  to  fill  in  tiie  window’s  data  fields.  TWo  order  types  take  only  a  single  argument: 
Deploy  inompts  for  a  destination  (the  syston  itself  selects  tiie  fi^iter)  arid  Recall  prompts  for  an  aircraft. 
Three  order  types  are  binary  -  Move  (airerttft,  destination).  Vector  (fighter,  threat)  and  Reftid  {aircraft, 
umker)  -  and  the  remaining  order  type.  Relieve,  is  ternary  (forward  fighter,  backtp  fighter,  tanker).  The 
Move  and  Deitioy  windows  also  include  a  cycle  item  for  choosing  the  aircraft  speed,  either  Slow  or  Fast. 
After  filling  in  all  requested  data,  a  final  didc  on  the  window’s  “Accept  Data”  button  (or  anywhere  on  the 
radar  screoi)  executes  the  order  and  a  didc  on  the  “Caned”  button  aborts  it. 

The  lesonUance  of  the  syntax  of  most  of  these  operatiems  to  that  of  NL  has  already  been  noted.  In 
pmticuUu;  saliences  of  the  type  illustrated  in  (1)  are  natural  equivaloits  to  the  first  five  grtqjhical  order 
types,  witfi  proper  names  cmutructed  using  the  arguments’  ID  numbers  serving  as  the  linguistic  analog  of 
direct  graphical  referoioes. 
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(1)  Deploy  afighur  to  station  5  ffast]. 

Recall  fighter  14. 

Move  fighter  14  to  station  5  [slow}. 

Vector  fighur  12  to  track  3. 

R^uel  fighter  14  at  tanker  1. 

Eucalyptus  also  accepts  the  sim(de  syntactic  variant  illustrated  in  (2).  The  grammar  also  permits 
fil^hters  to  be  Jdressed  directly  (3).  but  this  is  discouraged  since  it  nins  contrary  to  the  im{^cit  syntax  of 
the  grqihical  interface,  where  aircraft  are  the  subjects  of  rmters  rather  than  their  direct  recipiaus. 

(2)  Have  a  fighter  deploy  to  station  5. 

Have  fighter  14  return  to  the  carrier. 

(3)  Fighter  14,  return  to  the  carrier. 

Vector  to  track  3,  fighter  12. 

Rtfitel  at  tanker  1. 

For  each  of  these  verbal  inputs,  after  a  second  or  two  of  processing  time.  Eucalyptus  generates  a 
dialogue  window  just  like  the  (Mie  used  for  the  correqxmding  graphical  operation  but  with  the  verbal  argu- 
mertts  and  modifiers  already  filled  in.  allowing  the  user  to  chedc  the  input  data  for  accuracy  before  having 
the  order  executed.*  Final  acceptance  or  rejection  can  be  issued  either  verbally  {Okay,  Tha^s  fine.  Never 
mind.  Cancel  that)  or  graphically  with  a  single  mouse  click. 

Note  in  the  first  examine  of  (1)  that  the  appropHiate  NP  in  a  verbal  Dc|^y  operatitm  is  the  indefi¬ 
nite  afighur,  acknowledging  that  the  user  does  not  care  whidi  one  and  allows  the  system  to  choose,  as  it 
does  in  the  correqxMiding  grs^cal  operatitm  (since  in  the  KOALAS  GUI  fighter  icons  only  bectmie  visi- 
Ue.  and  thus  selectable,  tmce  diey  are  airborne).  Altemativdy.  in  Eucalyptus  the  user  can  name  a  specific 
fighter  known  to  be  tm  the  carrier  {Deploy  fighur  18  to  station  5).  If  the  tunned  fighter  is  not  on  tire  carrier 
but  has  already  been  ttepdoyed.  Eucalyptus  ignores  the  order  by  {unducing  no  dialogue  window  for  it 

Altmg  the  same  lines,  a  fighter  that  has  crashed  continues  to  exist  in  the  simulatitm  (an  entry  for  it 
remains  in  die  Hghler  Status  wiiKlow)  but  its  kxm  dis^ipears  from  the  grapdiical  disiday  with  the  appit^- 
ate  result  that  it  can  no  Itmger  be  selected  graphically  as  an  argument  to  a  fighter  or^r.  If  its  ID  numb^  is 
typed  into  the  dialogue  window’s  argumem  field,  however,  the  system  accepts  the  input  but  then  simpfiy 
disregards  the  order,  so  Eucalyptus  behaves  in  just  die  same  way  when  an  order  is  verbally  issued  to  a 
downed  fighter. 

Users  of  the  KOALAS  tool’s  grs^cal  interface  have  ctmiplained  about  the  number  of  low-level 
medumical  operatitms  it  takes  to  carry  out  an  aircraft  order.  For  example,  as  many  as  six  mouse  (^ratkms 
are  needed  to  issue  a  fighter  command:  one  to  bring  up  the  menu  of  order  types,  another  to  select  the  order, 
from  (me  to  three  to  specify  the  arguments  andfor  param^r  to  the  dialogue  window,  and  one  more 
to  ctmfirm.  While  this  could  certainly  be  considered  a  design  shortcoming  in  the  KOALAS  grapdiical  inter¬ 
face  that  could  be  remedied  by  some  less  mouse-intensive  grapdiical  interaction  te(dmi(iue,  most  users  of 
oommonly  availatde  GUI-bas^  rqppdications  cm  pmobaldy  drink  of  similar  siturdirms  they  have  experi¬ 
enced  at  (me  time  or  anothen^re  the  number  of  grrqdiical  irqmts  demanded  by  the  system  became  tiring 
and  mechanicaL  Users  of  an  NL  interface  must  provitle  all  the  same  parameters  to  ctmiidex  interactions  as 
in  the  gn^cal  interface,  but  the  ease  of  spokm  NL  certainly  seems  to  provide  a  pdeasant  alternative  to 
graphical  irqmt  (and  vice  versa:  one  may  tire  of  thinking  and  interacting  verbally  and  choose  to  swiftdi  back 
to  the  hand-eye  mode  of  grqrhical  interactitm). 


*Tlie  molivaikiai  bdund  diii  design  decision  « (tisouMd  later  m  the  Mctkm  “Multiinedu  Integration.*’ 
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Another  problon  that  arises  occasionally  in  window-based  graphical  interfaces  is  visibility  obscu¬ 
ration.  For  example,  during  a  Vector  operation  in  the  KOALAS  GUI,  the  popup  menu  and  dialogue  win¬ 
dow  can  obscure  the  destination  threat’s  icon,  making  it  impossible  to  select  with  the  mouse.  (Indeed,  the 
graphics  obscure  that  paiticular  region  of  the  radar  display  so  much  of  the  time  that  adversaries  playing 
scenarios  against  each  other  will  sometimes  have  their  enemy  raids  attack  from  that  bearing  so  that  they 
can  penetrate  ttieir  opponem’s  defenses  utuioticed!)  While  again  these  should  properly  be  consider^ 
design  shortcomings  in  the  KOALAS  interface,  the  fundamental  point  remains  that  in  any  window-based 
graphical  interface  where  much  or  all  of  the  visual  “real  estate”  of  the  screen  is  already  in  use,  newly  cre¬ 
ated  windows  will  necessarily  obscure  some  component  or  another  of  the  GUI  and  possiUy  require  resiz¬ 
ing,  moving,  hiding,  closing,  etc.  if  the  obscured  portion  of  the  display  needs  to  be  accessed.  SitKe 
arguments  are  provided  verbally  in  a  NL  interface,  however,  visilnlity  is  never  an  issue  (e.g.,  one  can  refer 
to  a  threat  even  if  its  icon  is  temporarily  obscured).  The  down  side  to  verbal  reference  is  that  greater 
demands  are  made  on  the  user’s  memory,  requirir^  that  s/he  be  able  to  ruune  or  describe  the  intended 
object,  which  is  not  the  case  with  direct  manipulation. 

Relieve 


Unlike  the  five  order  types  just  discussed,  Relieve’s  three  participants  do  not  corre^nd  in  num¬ 
ber  or  canonical  syntactic  order  to  the  thematic  role  arguments  of  the  corresponding  English  verb,  which  is 
strictly  Innary  (X  relieves  Y).  This  order  type  can  thus  be  thought  of  as  a  composite  of  two  simpler  actions. 
Relieve  and  Refkiel:  X  relieves  Y  so  that  Y  can  refuel  at  Z.  The  only  way  this  can  be  expressed  compactly 
in  English  is  by  using  an  embedded  infinitival  clause  whose  zeroed  subject  is  interpreted  as  coreferent  with 
relieve's  direct  object  (4). 

(4 )  Have  relieve  fighter  12  to  refuel  at  tanker  1 . 

SentetKe  (4)  illustrates  how  the  verb  relieve's  thematic  grid  gives  {ximary  attention  to  the  backup 
{fighter  14),  assigning  it  the  syntactic  role  of  subject.  In  the  grt^cal  interface,  however,  the  forward 
hotter  (the  one  being  relieved)  is  referenced  first  and  die  backup  fighter  second.  This  is  undoubtedly 
because  the  forward  fighter,  since  he  needs  refueling,  is  considered  the  primary  focus  of  interest,  with  the 
backiq)  {daying  a  supporting  role  and  the  tanker  (die  third  argument)  the  most  passive  role  of  all.  A  more 
dubious  Elfish  fdir^ng  (5)  is  required  to  capture  this  ordering. 

(5 )  Relieve  fighter  12  with  fighter  14  to  refuel  at  tanker  1 . 

Finally,  although  Eucalyptus  can  handle  both  (4)  and  (5),  to  handle  more  complex  forms  such  as 

(6)  would  require  the  addition  of  ad  hoc  techniques  for  coalescing  two  simple  sentences  into  a  single 
KOALAS  command. 

(6)  Have  fighter  14  relieve  fighter  12  so  he  can  refuel  at  tanker  1. 

Have  fighter  12  refuel  at  tanker  1  and  have  fighter  14  relieve  him. 

Since  the  Relieve  order  conflicts  with  the  verb  relieve  in  argument  stmcture,  better  integration 
b^een  graphics  and  NL  would  be  achieved  by  changing  the  graphical  Relieve  constma  to  a  binary  oper¬ 
ator  and  use  it  in  combination  with  the  existing  Refkrei  operator  to  achieve  Relieve’s  current  operational 
definitioiL  A  completed  Relieve  order  would  then  cause  a  Reftiel  dialogue  window  to  automatically  come 
up  widi  the  relieved  fighter  already  filled  in.  This  approach  would  allow  Eucalyptus  to  handle  all  of  (4-6) 
with  no  ad  hoc  fixes  at  all. 

Should  the  order  of  arguments  in  the  graf^cal  Rdieve  operation  then  be  reversed  to  match  the 
more  canonical  English  usage  of  (4)?  That  would  probably  be  of  little  value  since  the  less  common  usage 
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of  (S)  is  also  valid.  It  is  worth  noting,  however,  that  in  the  gn^cal  interaction  the  user  must  take  time  to 
stop  and  interpret  the  dialogue  window  slot  labels  (“Forward  Fighter”  and  “Backup  Fighter”)  to  determiiK 
which  one  represents  the  fighter  needing  refueling  and  which  the  relief.  This  is  not  the  case  with  verbaliza¬ 
tions  like  (4),  where  our  inherent  knowledge  about  the  verb  relieve  automatically  informs  us  which  argu¬ 
ment  is  which.  While  it  might  be  possible  to  revise  the  slot  labels  to  increase  their  recognizability,  the  fact 
remains  that  NL  syntax  and  semantics  provide  that  role  information  automatically  whereas  the  order  of 
arguments  in  a  dialogue  window  does  not. 

Underspec^  inputs 

Arguments  can  be  omitted  from  Eucalyptus  NL  irputs  just  so  long  as  the  resulting  utterance  is  still 
grammatical.  Example  (7)  shows  some  accept^le  Fighter  Management  orders.  In  each  case  a  dialogue 
window  comes  up  with  only  the  specified  data  filled  in,  prompting  the  user  to  enter  the  next  piece  of  miss¬ 
ing  information  using  the  mouse.  The  examples  in  (8),  however,  are  ungrammatical  (tl^  stigma  “*” 
denotes  a  syntactically  ill-formed  phrase)  and  are  rejected  by  the  NL  parser  with  the  message  “Sorry,  don’t 
understand.” 

(7)  Relieve  fighter  14. 

Have  fighter  14  refuel. 

Deploy  a  fighter. 

(8)  *Recall  to  the  aircrtrfi  carrier. 

*Have  fighter  1  relieve. 

There  are  a  number  of  reasons  why  users  might  issue  underspecific  orders  such  as  (7)  (see  Ref.  10 
for  a  related  discussion).  First,  they  might  have  forgotten  that  an  order  requires  a  particular  argument,  such 
as  the  Deploy  order  requiring  that  a  destination  secmr  be  specified.  Second,  they  might  prefer  to  initiate  the 
order  verbally  and  then  fill  in  certain  arguments  graphically.  For  example,  it  is  more  convenient  for  the  user 
to  simply  say  Deploy  a  fighter  and  then  specify  the  destination  wiA  a  quick  mouse  click  on  the  radar 
screen  than  to  have  to  mentally  recall  the  sector  ID  number  associated  with  a  particular  screen  location  and 
include  it  in  the  verbal  input  (Deploy  a  fighter  to  sector  5).  Hnally,  the  user  mi^t  assume  that  the  system 
will  automatically  fill  in  certain  missing  arguments:  for  example,  if  there  is  only  one  fuel  tanker  in  the  sim¬ 
ulation  scenario,  the  user  might  assume  that  R^el  fighter  12  is  sufficient  input  and  that  the  syston  will 
naturally  fill  in  the  ID  of  the  only  available  tanker.  Since  the  original  graphic^  interface  does  not  do  this, 
however,  neither  does  Eucalyptus,  to  preserve  semantic  equivalence  between  the  two  interface  media. 

Note  that  tte  user  cannot  bring  up  an  empty  dialogue  window  by  simply  uttering  the  verb  Deploy, 
because  the  NL  understanding  component  interprets  that  utterance  as  an  elliptical  imperative  telling  the 
addressee  (in  this  case  the  KOALAS  tool  itself)  to  leave  the  carrier,  which  violates  the  system’s  semantics. 
If  the  user  had  previously  been  addressing  a  fighter,  however,  that  fighter  becomes  the  implicit  subjea  of 
the  imperative  and  a  dialogue  window  would  come  up  with  the  fighter’s  ID  filled  in.  This  illustrates  how 
Eucalyptus  is  not  a  simple  speech  input  system  where  one  can  just  utter  the  names  of  widget  labels  (i.e.,  the 
menu  item  label  Deploy),  ^t  a  spoken  NL  system  that  obeys  at  least  some  of  the  rules  of  dialogue  dis¬ 
course. 


Threat  Assessment 

The  Threat  Assessment  menu  provides  four  action  types:  Make  (create  a  hypothetical  threat). 
Delete  (delete  a  threat).  Alter  (change  a  threat’s  attributes),  and  Lock  on  'D'ack  Cock  a  threat  onto  a  radar 
blip).  As  with  the  Fighter  Management  operations,  selecting  an  action  type  brings  up  a  dialogue  window 
whose  text  fields  can  be  filled  automatically  by  clicking  on  the  radar  screen  in  response  to  system  prompts. 
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Delete  and  Lodi  on  Thick  are  unary  operators,  prompting  only  for  the  threat  to  be  deleted  or 
lodced  (Xito  a  radar  blip  (the  blip  must  be  at  the  same  coordinates  as  the  threat  and  so  is  not  needed  as  a 
second  aigument).  Make  is  also  unary,  prompting  for  the  radar  screen  location  where  the  new  threat  is  to 
be  posititHied.  Alter  is  binary,  prompting  for  a  threat  and  a  new  position.  Except  for  alter,  the  correspond¬ 
ing  English  verbs  have  the  same  argument  structure  as  the  gn^cal  operations,  giving  us  the  basic  NL 
equivaloits  of  (9). 

(9)  Make  a  threat  here. 

Delete  threat  1. 

Lock  threat  •  r  track. 

As  with  Hghter  Management  orders,  these  fully  specific  utterances  bring  up  completely  data-filled  dia¬ 
logue  windows  ready  for  final  confirmation,  whereas  the  underspecific  utterance  Make  a  threat  brings  up  a 
dialogue  window  promiHing  for  the  required  position  argument. 

Alter 


Although  Alter  has  the  same  argument  structure  as  the  Move  filter  management  operation.  Move 
fighter  1  here  is  grammatical  but  *  Alter  fighter  I  here  is  not  The  reason  the  Alter  (^ration  is  so  named  is 
that  the  didogue  window  can  be  used  to  change  other  threat  attributes  beades  position.  When  the  user 
selects  a  threat  the  direat’s  current  attributes  (type  of  threat  number  of  aircraft  bearing,  distance,  speed 
and  course)  are  filled  in  automatically  as  defaults.  A  new  aircraft  type  and  speed  can  thoi  be  manually  cho¬ 
sen  from  cycle  items  in  the  window,  arxl  the  Number  of  Aircraft  text  field  can  be  edited  with  the  key¬ 
board.  If  tte  user  responds  to  the  “Qick  Left  Button  for  Position”  prompt,  the  threat’s  Bearing  arid 
Distance  fields  are  automatically  updated  to  reflea  the  new  position,  and  its  Course  field  is  updated  to 
reflect  the  new  default  heading  (towards  the  carrier).  A  new  course  can  then  be  assigned  by  clicking  the 
right  mouse  button,  which  causes  the  prompt  ”Qick  Left  Button  for  Course”  to  rqipear,  and  flie  position 
clidted  on  the  radar  screen  then  defines  the  threat’s  new  course. 

Like  the  Relieve  fighter  managonent  order,  more  complex  i^irasings  such  as  subordinate  dauses 
and  genitival  constructions  are  required  to  verbalize  both  the  bade  Alter  operation  (10)  as  well  as  the  other 
ways  that  a  hypothetical  flireat’s  characteristics  can  be  modified  (11).*  Th^  can  also  be  combined  in  vari¬ 
ous  ways  as  illustrated  in  (12). 

(10)  Alter  threat  1  to  have  this  position. 

Alter  threat  I  to  be  over  here. 

Alter  threat  1  to  be  located  here. 

Alter  threat  J’s position  to  here. 

(11)  Alter  threat  1  to  a  Badger. 

Alter  threat  1  to  be  5  Badgers. 

Alter  threat  I’s  bearing/^tancel course  to  3(X). 

Alter  threat  1  to  have  this  heading. 

Alter  threat  1  to  be  heading  here. 

Alter  threat  1  to  be  moving  fast. 

(12)  Alter  threat  1  to  five  Badgers  located  here  moving  here  fast. 

Alter  threat  1  to  be  heading  herefrom  this  position. 


*The' syntax  of  the  gr^dical  operation  does  not  allow  die  user  to  select  a  new  heading  for  a  dneat  widioin  first  (Te)selecting  die 
dmat's  position  as  well.  Since  die  NL  syntax  does  not  inqxxe  diis  constraint,  diis  siiggesu  that  die  graphical  opaadon’s  syntax 

should  be  revised  aoconhngly- 
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AU  the  ii^ts  above  are  considered  fully  specific  and  advance  the  dialt^e  window  to  its  final 
“dick  Left  Button  to  Accept”  prcMnpt,  since  die  threat’s  current  posititui  remains  the  default  if  no  new 
positicm  is  s^iecified.  The  underspecific  (13)  and  (14).  however,  leave  the  window  in  the  imennediate 
“Click  Left  Button  for  Position”  and  “Click  Left  Button  for  Course”  states,  respectively. 

(13)  Alter  threat  1. 

Alter  threat  Fs  positionibearingidistance. 

(14)  Alter  threat  Vs  headingl course. 

Finally,  since  die  Make  command  dialog  window’s  structure  and  behavior  is  virtually  identical 
to  Alter,  the  various  NL  capatrilities  outlined  above  transfer  easily  to  the  verb  make  as  well  (IS). 

(15)  Mcdce  2  Badgers  located  here  heading  here  fast. 

Make  a  pathfinder  M>ith  a  course  of 300  at  this  position. 

Check  Boxes 

ChedL  boxes  (gra{diical  on/off  switches)  toggle  between  on  and  off  with  each  mouse  didt,  dis- 
[daying  a  check  mark  when  in  the  on  state.  The  KOALAS  tool  GUI  has  seven  check  boxes,  labded  Real 
ThKats,  Ground  IVuth,  Aircraft  lyails.  Radar  Trails,  Diagnostics,  Data  Collection,  and  Auto  Accept. 
The  first  four  control  the  display  of  information  on  the  radar  screen;  the  fifth  adds  rule-diagnostic  informa- 
titm  to  the  system’s  tactical  advice  explanations;  the  sixth  activates  cm-line  data  coUectimi;  and  the  last 
switches  the  system  into  automatic  advice  acce{Xance  mode,  defined  earlier. 

Natural  language  equivalents  for  these  were  determined  by  first  sdecting  a  verb  to  represent  the 
underlying  functionality  of  each  switdi.  For  the  first  five  switches  (all  labeled  with  just  noun  pluases),  tire 
imididt  operational  verb  chosen  is  show  or  display.  The  operational  verb  for  the  sixtii  is  just  the  denmni- 
nalization  collect  of  the  switch’s  label,  and  the  label  of  the  sevoith  switch  is  already  a  verb  (albeit  fabri¬ 
cated),  asito  accept. 

All  these  verbs  are  strictly  transitive,  so  by  adding  tire  label’s  noun  phrase  (or,  in  the  case  of  Auto 
Accqrt,  the  noun  advice)  as  dired  objed  we  gd  the  simplest  NL  equivalents,  (16).  Nominalizatums  of 
eadi  of  these,  introduced  by  the  auxiliary  do  (17),  are  also  accqded,  and  an  explicit  don’t  auxiliary  trans¬ 
forms  each  of  (16-17)  into  its  complementary  operation  (18).  Alternatively,  ea^  of  (16-17)  can  he  made 
progressive  and  introduced  by  the  verbs  start  or  stop  to  switch  on  or  off,  respectively  (19).  Finally,  the 
swireh  label  can  simply  be  prefixed  by  the  goieric  verb  turn  onJoffiTD).  In  eadi  case,  the  NL  command 
both  changes  the  intenial  state  of  the  KOALAS  fnxrcess  and  goierates  the  appropriate  visual  feedbadc  in 
tire  graphical  interface  (i.e.,  adds  or  subtracts  the  check  mark). 

(16)  Show  real  threats  (ground  truth...). 

CMectdcm. 

Auto  accept  advice. 

(17)  Do  real  threat  display. 

Do  data  collection. 

Do  auto  accqttance. 

(18)  Don’t  show  real  threats. 

Don’t  collect  data. 

Don’t  auto  accept  advice. 

Don’t  do  real  threat  display  (data  collection,  auto  accq>tance). 
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(19)  Start  showing  real  threats. 

Stop  collecting  data. 

Start  auto  accepting  advice. 

Sugp  doing  real  threat  display  (data  collection,  auto  acceptance). 

(20)  Turn  on  real  threats. 

Turn  off  data  collection. 

Turn  on  auto  accept. 

Whereas  a  mouse  elide  just  toggles  the  cunent  state  of  the  switch  to  its  crnni^anait,  each  NL 
omunand  ejqsnessly  declares  die  goal  state  of  the  operadon.  If  the  switch  is  already  in  ttiat  state.  Eucalyptus 
inftmns  the  user  of  the  fact  with  the  messages  “Already  eta”  or  “Already  off**  radier  than  simfdy  perfoim  no 
operatitm.  Start,  su^,  turn  on,  and  turn  off  9st  modeled  as  presuppositional,  that  is,  both  they  and  their 
n^adons  are  interpreted  as  implying  the  troth  of  their  aigument  Thus,  if  data  collection  is  currendy  off. 
Eucalyptus  will  re^nd  “Already  off”  in  all  the  cases  illustrated  in  (21). 

(21)  Turn  off  data  collection. 

Don’t  turn  off  data  collection. 

Suyt  doing  data  collection. 

Don’t  sug>  doing  data  collection. 

Cycle  Chidce  Items 

The  Experimenter  Control  panel  contains  two  cycle  choice  items.  Zoom  and  Speed,  which 
re^)ectively  alter  the  radar  screen  magnificadtm  and  the  ratio  of  Emulation  time  to  real  time.  A  range  of 
veits  were  diosai  to  model  these  choice  items:  zoom  (in/out),  make,  set,  slow  downlspeed  up,  change,  and 
increase/decrease,  illustrated  in  (22).  As  with  the  check  boxes,  the  NL  command  both  changes  die  state  of 
the  coire^xMiding  KOALAS  variable  and  updates  die  visual  di^ay  to  reflect  the  new  value. 

(22)  Zoom  out. 

Zoom  in  to  a  magnification  of  4. 

Make  the  screen  magnffkadon  3/2. 

Set  the  magnification  to  2/3. 

Slow  down  the  simulation. 

Speed  the  simulation  up  to  2  times  real  time. 

Make  die  simulation  speed  real  time. 

{  Change  /  Increase  /  Decrease  }  the  ^ed  to  20. 

{  Change  /  Increase  /  Decrease  }  the  magnification. 

hi  die  gn^cal  interface,  cliddng  rqieatedly  on  a  cycle  item  iiKiements  its  value  tqiwaid  to  die 
maximum  value  and  dien  cycles  badt  to  the  lowest  value,  so  goal-less  verbal  inputs  like  Zoom,  Change  the 
screen  magnification,  and  Change  the  simulation  speed  do  the  same.  Most  other  phrasings,  however,  have 
inesuppoddons  diat  must  be  satisfied  before  execution.  For  example.  Zoom  in  presupposes  that  the  cunent 
magni&»tkm  is  less  dum  die  maximum  value,  and  Zoom  in  to  1  presupposes  that  it  is  curraitly  less  than  1 . 
If  such  presiqiposititHis  are  not  met.  Eucalyptus  r^xmds  “Less/Greater  than  that  now,”  “At  diat  value 
now,”  or  “At  maximum/hiinimum  value  now.”  The  graphical  interface  does  not  i»otest  if  die  user  resets  a 
parameter  to  its  cuimit  value,*  however,  so  die  verbs  zoom  aixl  nialx  are  not  treated  as  presuppoational, 
mddng  sentences  like  Zoom  ro  4  and  Make  the  simulation  speed  2  accqitatfle  r^ardless  of  the  parameters  ’ 
cunent  values. 


♦Rweahit  (he  wraen  megdficetion  lo  its  cuinitt  tnloe  is  one  wi^  of  rednwing  the  soecn  to  clear  away  grajihical  gjkdies. 
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Buttons 

The  six  domain-functional  buttons  in  the  interface  are  labded  Next  Advice,  Explain  Advice, 
Remove  E:q>lain,  Accept  Advice,  Suspend/Resume,  and  Exit  Simulation.*  The  labels  of  all  of  these  but 
the  first  (which  Eucalyimis  assigns  the  implicit  verb  show)  include  a  verb  refnesenting  the  underlying  ccnn- 
mand.  As  with  the  check  box  commands.  Eucalyptus  requires  that  ail  the  button-equivalent  veite  be  used 
transitively,  as  in  (23). 

(23)  Show  the  next  advice. 

Explain  the  [current]  advice. 

Remove  die  [advice]  explanation. 

Accept  the  [current]  advice. 

[  Suspend  I  Resume  )  the  simulation. 

Exit  [from]  the  simulation. 

In  addition  to  these  six  buttons,  each  fighter  management/threat  assessmoit  dialogue  window  pro¬ 
vides  two  buttons  labeled  Acce|d  Data  and  Caned  for  confirming  or  cancelling  the  currently  proposed 
Older.  As  before.  Eucalyptus  accepts  transitive  sentences  (Accept  the  data.  Cancel  the  (deration)  for  these 
two  operations.  In  adtUtion,  since  the  dialogue  window  interaction  takes  the  form  of  a  “subdialogue" 
embethled  in  die  top-level  user-machine  dialogue,  formulaic  elliptical  inputs  like  Okay  and  Never  mind  are 
also  accepted. 

Radar  Screen  Mouse  Opo^tions 

As  mentioned  earlier,  mouse  clicks  on  the  radar  screen  have  predefined  functionality  in  the 
KOALAS  giaifiiical  user  interface:  a  left  clidk:  on  an  aircraft  icon  displays  its  position  information,  a  mid¬ 
dle  dick  recenters  die  radar  screen,  and  a  right  elide  rni  an  aircraft  shows  any  direat  si^itings  (tallies)  asso- 
dated  widi  it  We  will  consider  each  of  these  in  turn. 

The  description  of  mouse-left  in  the  user’s  manual  is  “Display  information  about  an  aircraft’s  posi¬ 
tion.’’  A  verbal  input  along  those  lines  seems  too  wordy  to  make  NL  an  accqrtaUe  alternative  compart  to 
the  simffiidty  of  a  single  mouse  click,  so  Eucalyptus  instead  interprets  mouse-left  amply  as  the  verbs 
describe  or  tell,  accqiting  sentences  such  as  (24). 

(24)  Describe  that  fighter. 

Tell  me  about  threat  1. 

Mouse-generated  aircraft  position  popup  windows  automatically  vanidi  once  the  mouse  button  is  released, 
a  form  of  direct  manipulation  in  whidi  the  user  in  effect  “holds  up”  die  di^lay  window  with  the  mouse  for 
viewing  and  then  “drops”  it  when  through  examining  it,  giving  mouse-left-down  the  sanarnics  “describe” 
and  mouse-left-up  the  semantics  “ranove  description.”  Correspmidirigly,  a  window  generated  by  a  verbal 
describe  remains  on  the  screen  until  expliddy  dismissed  (Okay,  Remove  the  descripdori).  This  parallels  an 
undocumoited  “pustqxn”  behavior  already  |xesem  in  the  gri^cal  interface:  moving  the  mouse  over  the 
window  before  releasing  the  button  leaves  the  window  stuck  to  the  screen  until  dismissed  by  a  second  left 
didt  (m  an  empty  portion  of  the  screen  (if  mote  than  one  posititm  window  is  pinned,  only  the  most  recent 
(me  is  dismissed  -  the  others  must  be  killed  from  their  tidebar  menus).  As  a  result.  Eucalyptus  allows  ver¬ 
bally  goierated  windows  to  be  dismissed  by  a  second  left  click  as  welL 


*Four  ofliCT  bunoitt  in  die  gra|ihical  intafiKe  (Fighter  MaaagciBeBt,  Threat  AaMSBmeBt.  Experimenter  Contra!,  nd  Qnit) 

ecrae  only  to  bring  iq>  or  dinniss  other  gnqphical  enddet  like  menus,  popup  windows,  or  control  pends.  As  duenssed  eariier,  Euce- 
bptns  does  not  accept  NL  iiqwt  for  such  “metslinguisdc”  grqdiical  opersdons. 
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Mouse-middle  simjdy  translates  in  Eucalyptus  to  commands  of  the  fonn  Center  the  (radar)  screen 
here,  where  “here"  will  be  discussed  later  in  tbe  section  "Deictic  Reference."  Mouse-right  behaves  in 
stnnewhat  similar  fa^on  to  mouse-left,  the  icons  representing  threat  sightings  remaining  visible  on  the 
soemi  until  dismissed  by  a  sectmd  right  mouse  click  at  an  empty  screen  position,  but  in  this  case  die  accu¬ 
mulated  disf^ays  of  successive  tallies  are  all  erased  at  once.  Examine  (25)  illustrates  the  corresponding  NL 
(XHnmands. 

(25)  Show  the  [threat]  ( tally  /  sightings  )  for  UAV 16. 

Remove  the  [threat]  ( tallies  /  sightings  ). 

NL  INTERFACE:  BEYOND  GRAPHICS 

So  far  in  this  report  we  have  looked  at  simple  NL  sentences  that  represent  direct  translations  of 
individual  GUI  command,  thus  constituting  only  a  change  of  interface  medium.  The  goal  in  Eucalyptus 
was  not  to  simply  produce  a  multimedia  interface,  but  to  exploit  the  additional  expressive  power  th^  NL 
provides  above  and  beyond  what  is  possible  in  the  gn^Aiical  interface,  while  still  keeping  the  two  fuUy 
integrated  (multimedia  integration  is  discussed  in  the  r^xt  major  section).  Three  features  of  NL  that  most 
distinguish  it  frcnn  direct  manipulation  ate  its  capability  for  parai^irase  (synonymy  and  variant  syntax),  set- 
theoretic  and  logical  operations  (particularly  denotative  reference),  and  context  mainterumce  (discourse 
tracking).  Since  our  group  had  already  explored  the  first  of  these  in  the  Interims  project  [7],  the  remainit^ 
two  were  chosen  for  investigation  in  Eucalyptus  atul  are  discussed  in  the  sections  that  follow. 

Referoice 

NL  reference  allows  a  user  to  specify  cognitively  significant  sets  of  objects  descriptively  through 
the  use  of  attributive  and  relational  predicates,  quantifiers,  and  logical  operators.  This  is  considerably  dif- 
ferait  from  the  direct  manipulation  paradigm,  in  which  operations  ate  applied  to  individually  selected 
objects.  Descriptive  denotation  also  makes  possible  mote  efficioit  information  access  in  the  form  of  NL 
queries,  the  user  asking  about  the  e«stence  or  identity  of  cognitively  interesting  sets  of  objects  or  the  val¬ 
ues  of  their  attributes;  the  Eucalyptus  query  facility  will  be  described  shottiy. 

NL  referring  expressions  in  Eucalyptus  can  be  prmiouns  (it,  he,  diem),  headless  quantifiers  (one, 
any),  locative  adverbs  (here,  there),  proper  names  (figluer  1),  or  noun  {riirases:  indefinite  (on  F14,  any  of 
the  searching  UAVs,  no  fighter),  defitiite  (that  tanker,  the  fighters  moving  to  station  2),  or  universal  (all  the 
fighters  on  the  carrier,  every  UAV).  NP  ptemodifiets  can  be  nominal  (radar  trails),  adjectival  (hypodietical 
threat),  verbal  (deployed  fighter,  moving  F14),  or  genitival  (fighter  I’s  missiles);  postmodifiers  are  either 
prepositional  i^itases  (an  aircrttft  on  the  carrier,  the  bearing  of  fighter  1)  or  relative  clauses  (a  threcu  being 
vectored  to,  the  fighters  [that  are]  low  on  fuel).  NPs  can  be  conjoined  conjunctively  (the  tanker  and  all  the 
fighters,  fighter  14  and  tanker  I)  or  disjunctively  (either  an  F14  or  one  of  the  UAVs,  UAVs  3, 4,  or  6)* 
Indefinite  NPs  are  interpreted  intensionsdly  only  in  the  context  of  tiie  tiireat  management  commands  Make 
and  Alter  (Create  a  Badger  here.  Change  threat  1  to  a  Badger)  and  ate  inteti»eted  as  extensional  refer¬ 
ences  in  aU  other  cases. 

Nominal  and  adjectival  NP  premodifiers  are  interpreted  as  object  specifiers,  e.g.,  radar  trails  and 
aircrtfl  trails  refer  to  two  differmrt  domain  entities  that  together  could  be  referenced  as  all  the  trails.  All 
other  {nemodifiers  and  postmodifiers  are  interpreted  as  relational  constraints  to  be  diecked  against  the 
KOALAS  internal  datal^  when  computing  die  referent  set,  e.g.,  die  moving  fighters  is  the  subset  of  fi^- 


*Diie  10  amtraintt  in  die  logical  fonn  genenuor  (desoibed  later),  conjoinings  that  combine  conjunction  and  disjunction  (other  all 
the  fighters  or  none  of  the  UAVs)  are  not  supported. 
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ers  that  are  moving  somewhere,  equivalent  to  the  fighters  moving  to  a  tanker  or  a  station.  Postmodifiers  in 
particular  are  fuUy  iterative  and  recursive  (26). 

(26)  a  threat  located  here  consisting  cfS  Badgers 

every  fighter  moving  to  the  station  that  fighter  2  is  holding 

Genitives  like  fighter  Vs  nussiles  are  inteipreted,  like  the  missiles  of fighter  1  or  the  missiles  fighter 
1  has,  as  references  to  the  values  of  aircraft  attributes  like  bearing,  distance,  course,  speed,  pounds  of  fuel, 
and  number  of  missiles.  Sudi  NPs  are  not  dereferenced  extensionally  but  are  treated  as  intmisional  formu¬ 
lae  and  evaluated  as  function  calls  during  execution  of  the  logical  form.  For  example.  Does  fighter  1  have 
a  bearing  of  301  goierates  a  logical  form  along  the  lines  of 

(TELLIF  (EQUAL  : THEME 

(ATTRIBUTE-VALUE  : PATIENT  'FIGHTER-1  : THEME  'BEARING) 
:GOAL  30)) 

where  the  attribute-value  function  call  returns  the  current  value  of  fighter  1  ’s  bearing  (obtained  by 
(xmsulting  the  KOALAS  tool’s  internal  state),  which  is  fiien  tested  against  the  goal  value. 

The  domain  objects  that  serve  as  linguistic  referents  are  generated  in  three  {biases.  Scenario-inde¬ 
pendent  entities  (KOALAS,  the  operator,  the  aircraft  carrier,  fighter  assignment  stations,  display/simula¬ 
tion  control  parameters,  and  generic  aircraft  attributes)  are  defined  in  an  external  Eucalyptus  knowledge 
base  loaded  at  system  startup.  Next,  Eucalyptus  creates  all  the  scenario-dqrendent  objects  (the  friendly  air¬ 
craft  and  actual  threats)  at  the  beginning  of  die  run  using  information  contairred  in  the  internal  data  struc¬ 
tures  that  KOALAS  generates  from  the  input  scetario.  Fmally,  Eucalyptus  represents  efdiemeral  objects 
(the  hypothetical  threats)  by  temporary  objects  created  by  consiiltmg  the  KOALAS  internal  database  on  an 
as-refereiKed  basis,  since  the  KOALAS  entities  those  objects  rqnesent  are  created  and  destroyed  dynami¬ 
cally  throughout  die  simulation  rua  Eucalyptus  detects  and  rqports  refermce  resoludon  failures  (specifier 
and  number  mismatches)  with  the  messages  “{No  I  Only  one  I  More  than  one)  sudi  <object  type>  exists.’’ 
Processirtg  of  the  ocMnmand  or  query  ceases  at  that  poirtt  unless  other  parses  are  availaUe,  in  whi(di  case 
the  syston  lotdts  for  an  alternative  interpretaticm  that  is  free  of  reference  errors. 

It  was  noted  earlier  that  one  feature  of  the  KOALAS  griq)hical  interface  is  to  only  permit  fighter 
management  and  threat  assessment  operations  to  be  performed  one  at  a  time  on  individual  arguments,  and 
that  no  sudi  operatitm  can  be  initiated  wMle  another  is  still  pending.  To  parallel  this  bdiavior  in  Eucalyp¬ 
tus,  verbal  fighter  or  threat  commands  containing  {dural  arguments  (e.g..  Lock  threats  I  and  2  on  track) 
Ixing  tq>  each  popup  confirmation  window  oim  at  a  time,  waiting  for  the  currmit  one  to  be  dismissed  before 
displaying  the  next  one.*  Since  die  user  may  wish  to  cancel  the  entire  series  of  dialogue  windows  that  can 
re^  from  such  an  utterance,  verbal  cancellation  expressions  like  Never  mind.  Forget  it,  and  Cancel  diat 
cancel  the  oitire  verbal  order,  not  just  die  current  in^vidual  window.  Eucalyptus  does  not  detect  applica- 
tkm-^)edfic  (M'agmatic  errors  involving  plural  arguments,  as  in  Move  fighter  1  to  stations  4  and  5  (a  fighter 
can  (My  move  to  one  station  at  a  time),  but  simply  generates  two  dialogue  windows  in  sequoice,  the  latter 
(if  confirmed)  overturning  the  effect  of  die  former  (if  confirmed). 

The  variety  of  referring  expressions  available  in  Eucalyptus  serves  two  main  functions.  One  is 
amqdioiic  reference,  to  be  discussed  in  a  following  sectioa  The  other  is  descriptive  refmence,  which 
allows  the  user  to  dmiote  (Hie  or  a  set  of  objects  by  didr  attributes  and  relationships,  relegating  the  work  of 
identifying  die  particular  individuals  to  the  system  itself.  Example  Q7)  is  a  ccHiiniand  that  a  KOALAS  user 
might  have  in  mind  and  could  execute  dir^y  as  NL  irqwt  to  Euctdyptus,  letting  the  syston  (dwose  a 
figltter  from  the  appropriate  set  and  identify  the  particular  threat.  Using  the  graphical  interface,  the  operattH 


*T)ie  MOM  bdisvior  occurs  in  cogoined  iiiq)en^ves,  e.g..  Create  a  rAreor  Aer«  OHUocifc  k  on  rrocL 
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would  have  to  first  interact  widi  the  system  in  various  ways  to  obtain  die  ID  numbers  of  the  oitities  in 
questkm,  aiWtrarily  choose  (me  of  the  filters,  find  the  selected  fighter  and  threat  (m  the  radar  screen,  and 
finally  dick  on  each  of  diem  with  the  mouse. 

(27)  /  want  to  vector  one  of  the  fighters  holding  station  1  out  u>  the  same  threat  fighter  2  is  vector¬ 
ing  to. 

Query  Fwility 

So  far  we  have  (mly  discussed  imperative  NL  sentoices  for  issuing  commands  to  die  system. 
Eucalyptus  also  indudes  a  <)uesdon-answeiing  facility  for  (pierying  the  internal  state  of  the  simuladon.  The 
query  ccmipcmem  handles  yesi/ho  quesdtms  (Are  any  fighters  moving?),  Wh-iiuestions  (Who  is  moving?) 
and  Wh-Noun  questi(ms  (Which  fighters  are  moving?).  Wh-<]uestions  are  restricted  to  asking  who,  what, 
where,  and  how  many  (much)  since  neither  KOALAS  nor  die  NL  [Hocessor  r^resents  time  history  (when) 
or  metaknowledge  (why,  how). 

bi  keeping  with  the  dedaon  to  treat  the  existing  graphical  interface  as  embodying  an  implidt 
spedficadtm  of  the  syston’s  intended  I/O  accessibility.  Eucalyptus  only  allows  the  user  to  query  for  that 
system  infi^rmadon  also  available  via  the  gnqihical  di^ays.  The  giiqMcal  interface  i»esents  simulatkm 
state  informadcm  in  three  ways.  First,  fighter  and  threat  status  is  continuously  di^ayed  in  tabular  foim  in 
text  output  windows  wlddi  user  can  scroll  dirou^  in  search  of  desired  iiiformatioii.  For  each  finder  in 
die  simulatkm,  the  Fighter  Status  window  diqdays  its  ID,  number  of  Sidewindor  missiles,  number  of 
Phoenix  missiles.  Bearing,  Distance,  Course,  pminds  of  Fuei,  Action,  and  Destination.  For  each  sus¬ 
pected  enemy  raid  that  is  currendy  hypothesized,  the  Threat  Status  window  shows  its  ID,  Number  of  air¬ 
craft,  number  Escaped,  number  Des^yeci,  Bearing,  Distance,  and  type. 

Second,  holding  the  left  moose  down  (m  a  fighter  or  threat  icon  on  the  radar  screen  brings  up  a  so- 
called  ”Uip  pc^up”  window  disidaying  aircraft  ID,  l^pe.  Bearing,  Range,  Status  (fighters  only),  and 
Nurnbor  and  Couiw  (direats  only).  Status  is  die  same  as  Actitm  +  Destimthm  in  the  Finder  Status  win- 
dow,  consisting  of  simide  text  strings  like  “Msctor  Tmck  3,”  “On  Carrier,”  and  “Holding  Sddion  1.” 
Rnally,  fighter  iccms  change  color  to  indicate  certain  state  changes:  normally  colcxed  cyan,  the  ic(ms 
change  to  yellow  when  vectoring  to  a  threat  and  green  when  (wt  of  weapons. 

Eucalyptus  uses  a  tradidtmal  database  qsery  system  iqi[xoach,  translating  each  question  into  a  per- 
formative-maAed  (joantified  logic  expresatm  whi(^  whoi  evrduated,  accesses  the  KOALAS  intetiud  data 
stractures  md  retrieves  a  s^  of  conforming  domain  ob^cts.  Hie  perftirmative  then  uses  this  result  to  com¬ 
pose  a  Idief  but  infiirmative  answer  qipropriate  to  die  way  the  qpiestion  was  formulated,  and  diqdays  diis 
answer  as  a  text  string.  Fbr  exam{de,  a  yes/ho  (piestitm  involving  a  universal  NP  (Are  all  the  threats  being 
vecuned  to?)  just  answers  “Yes"  if  no  excqdions  exist,  but  does  ntd  simidy  answer  “No"  otherwise  -  it 
goes  (m  to  enumerate  erther  the  positive  or  negrdive  instances  (depending  cm  which  set  is  shorter),  e.g., 
'*No,  an  but  Badger  #1  and  PadifiiKter  #2,”  or  “None"  if  time  are  only  excqdicms. 

Eucalyptus  composes  eUiptical  rather  dian  ccdnpletB  sentmices  as  reqxmses  dmidy  because  NL 
generation  is  not  an  issue  being  addressed  here.  Elliptical  reqxmses  result  in  a  less  mech^cal  and  more 
convosaiionttl  st]de  of  dialogue  and  also  sug^st  the  permissibility  of  eUiptical  NL  irqmts  by  the  user  [11], 
ffiacussed  in  die  next  secticm.* 


*Atnesmelinu^liowevei;d%ticainqionMiiniy  not  provide  suCSciemfeedbeditDreasstse  die  user  that  die  fystemoonectiy 
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As  mentioned  eailier,  aircraft  attributes  are  not  modeled  as  extensional  objects  but  as  intoisional 
formulae  whose  values  are  accessed  ftom  KOALAS  itself.  Eucalyptus  {Helixes  each  returned  attribute 
value  with  the  aircraft’s  ID  and,  if  necessary,  the  attrilHite  ty{)e.  For  exam{)te,  the  answer  to  the  query  What 
bearing  and  distance  do  the  moving  fighters  have?  prefixes  both  the  aircraft  ID  and  the  attribute  type  to  the 
result  values: 

F14  #1:  Bearing:  200  &  Distance:  300  and  F14  #2:  Bearing:  300  &  Distance:  400. 

Eucaly{)tus  does  not  do  any  syntactic  sugaring  to  the  string  values  that  KOALAS  returns  to  attribute  (pie¬ 
ties,  but  re{X)Tts  them  in  just  the  same  format  as  they  a{){)ear  in  the  grtqihical  display. 

The  advantage  of  a  NL  infMit  tpiery  facility  over  oxiventional  grai^cal  access  and  data  display 
technitpies  (as  well  as  more  advaiu^  gra{)hical  query  systems  [12])  is  similar  to  the  advantage  of  descri[>- 
tive  referring  expressions:  the  user  can  Ascribe  die  desired  information  and  let  the  system  {lerfoim  the 
search  operations  itself.  In  certain  cases,  however,  it  is  undoubtedly  more  convenient  to  visually  search  the 
talHilar  and  slot-filler  forms  of  the  KOALAS  graphical  display,  such  as  cpiickly  scanning  all  the  informa¬ 
tion  relative  to  any  one  individual  aircraft.  Thus  an  integrated  interface  like  Eucaly{)tus  gives  die  user  the 
option  of  either  mode  of  access,  depending  on  the  complexity  of  the  informati(Hi  being  sought 

Discourse  Capabilities 

The  goal  of  adding  discourse  tracking  to  die  Eucaly{)tus  human-c(Hn(xiter  dialogue  is  to  allow  the 
user  to  employ  vague,  ambiguous,  or  abbreviated  inixits  which  the  system  can  then  resolve  in  terms  of  the 
current  ditdogue  context  reducing  the  amount  of  repetition  in  user  input  and  increasing  the  naturalness  of 
the  human-machine  dialogue.  Eucalyptus  attempts  to  resolve  two  forms  of  abbreviated  NL  input  ana|4iora 
and  elliiisis.  Broadly  defined,  anaf^rs  are  abbreviated  references  to  entities  already  mentioned  earlier  in  a 
discourse.  They  are  one  of  the  most  ctnnmonly  used  NL  mechanisms  for  avoiding  excessive  wordiness  and 
keejping  a  discourse  focused  on  irew  rather  than  old  informatioa  Ellipsis  is  an  elision  of  lexical  material 
that  leaves  a  sentoice  or  {dirase  syntactically  incomplete  and  inteipreft^le  only  in  the  context  of  prior  utter¬ 
ances.  Although  naive  users  often  believe  that  comimters  with  NL  ca{>ability  actually  prefer  ablneviated 
(or  so-called  “telegraphic”)  in{)uts  of  this  son,  it  is  really  just  the  0{)fX)site:  the  less  complete  the  syntax  and 
semantics  of  the  in()ut,  the  more  inferential  work  tire  machine  generally  has  to  do. 

Amphora 

The  ana[^oric  referring  expressions  handled  by  Eucalypxtus  are  pronouns  (it,  he,  they),  headless 
quantifiers  (one,  any,  each),  demonstratives  (that,  those,  there),  and  reduced  definite  NPs  (the  Badger). 
Since  definite  NPs  are  not  necessarily  anaq^ric  but  could  be  references  to  entities  assumed  to  be  already 
known  to  the  user,  EucalyiHus  takes  a  standard  approach  [13]  of  trying  the  an^^ric  reading  first  and  dren, 
if  that  fails,  the  non-anaphoric.  For  example,  the  threat  has  a  non-anaqdioric  inteipretation  so  long  as  only 
(Hie  threat  exists  in  the  simulation,  but  has  only  an  anai^ric  inteipretation  once  additkHial  threats  apfrear. 
The  reference  resolution  mechanism  is  discussed  in  more  detail  in  a  later  section,  but  in  a  nutshell  is  a  sin- 
gle-c(Hitext-s{)ace,  focus-based  approach  that  searches  ordered  lists  of  data  structures  called  Discourse 
Oitities  that  are  generated  for  each  referring  expression  in  the  user’s  in{)ut 

There  is  little  oiportunity  in  the  KOALAS  domain  to  use  anaqdiora  in  {laired  im{rerative  clauses, 
because  for  the  most  part  the  application  task  only  retpiires  the  KOALAS  user  to  issue  an  order  to  an  air¬ 
craft  or  modify  a  hyiwthetical  threat  and  then  make  no  further  reference  to  tire  aircraft  or  fiireat  until  con¬ 
siderably  later  in  the  simulatitHi  run  (after  the  first  order  has  been  successfully  carried  out).  'The  (Hie 
excqHkm  is  the  pair  of  threat  assessment  operations  Make  and  Lo(&  on  TVardt,  which  are  commonly 
issued  in  se(]uaice  and  can  be  conveniently  verbalized  in  Eucalyiitus  as  Make  a  new  threat  here  and  lock  it 
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OH  track.  Eucalyp^  reqxMids  to  diis  input  by  creating  a  Make  dialogue  window  and  then  (after  it  has  been 
completed  and  dismissed  by  the  user)  a  Lock  on  TVack  window  with  the  ID  of  the  newly  created  thretu 
automatically  filled  in.  The  other  way  anaphora  might  occur  in  KOALAS  wmild  be  if  the  user  chose  to 
issue  a  sing^  ctmunand  or  query  using  more  than  one  sentrace,  as  in  (28).  but  merging  multii^  seiaences 
into  a  single  interface  conunand  m'  query  is  beymid  Eucalyptus’s  curratt  capal^ties. 

(28)  That  fighter  moving  to  station  1?  Have  him  rtfitei 

Relieve  fighter  I  to  r^fitel.  Have  fighter  12  relieve  him.  He  should  refuel  at  tanker  1. 

Make  a  threat  here.  It  should  consist  of  5  Badgers.  Give  it  a  course  of 300. 

Where  amqdioiic  reference  becomes  more  useful  is  in  connection  with  the  query  facility,  where  it 
can  enhance  the  sense  of  ct^rent  dialogue  between  human  and  machine,  for  example: 

Are  any  fighters  vectoring  to  threats? 

Yes.F14«l. 

Which  one  is  he  vectoring  to? 

Badger  #5. 

Are  there  any  fighters  still  on  the  carrier? 

Yes.F14#18andF14#]9. 

Deploy  one  ofdtem  and  vector  him  out  to  that  threat. 

The  user  can  refer  antqrhorically  to  entities  reported  by  the  query  answering  system  (e.g.,  he,  them,  and  that 
threat  refer  to  F14  #1.  F14s  #18-19.  and  Badger  #S,  respectively)  because  Eucalyptus  creates  a  Discourse 
Entity  for  each  ncnm  i^irase  generated  by  the  query  system  and  enters  it  into  the  discourse  histtny,  just  as  it 
does  for  eadi  referring  expressitm  uttered  by  foe  user. 

It  was  mentioned  earlier  that  Eucalyptus  does  not  ccmsider  KOALAS-initiated  events  sudi  as  the 
hypothesizing  of  a  new  foreat  or  foe  ptt^rosal  orexecudtm  of  tactical  advice  to  be  ccHitributions  to  foe  user- 
madiine  discourse.  As  a  result,  the  newly  tqrpearing  threat  ictm  or  the  entities  involved  in  foe  advised 
finder  order  cannot  be  refsraiced  anqfoorically.  For  extenple,  if  the  user  irttroduces  a  new  hypothetical 
foreat  and  foe  system  subsequently  introduces  one  of  its  own,  a  followup  anaffomic  reference  to  four  Areat 
co-refers  to  foe  user-  rafoer  than  machine-created  one. 

Rnally,  since  the  ^»edi  recognition  compmient  cannot  accommodate  rdative  clauses  due  to  limi¬ 
tations  (m  the  size  and  comifiexity  of  grammar  that  it  will  asxept,  a  relativized  imperative  like  Have  all  the 
fighters  vectoring  to  a  threat  that  are  low  on  fuel  return  to  the  carrier  must  eitiier  be  typed  in  or  else 
tqibrased  as  a  sequence  of  queries  and  final  imperative  employing  anaffoora: 

Which  fighters  are  vectoring  to  a  threat? 

F14#l.F14#3andF14#6. 

Are  any  of  them  low  on  fuel? 

Yes,F14#l. 

Have  Mm  return  to  the  carrier. 


Eucalyptus  handles  elliptical  followups  (eitiier  qtmries  or  imperatives)  by  using  the  {Mior  utter- 
nce’s  prdogical  form  (a  case  fnune  instantiation)  as  a  tonplate  to  be  re-evahiated  after  possittiy  rqtiadng 
one  of  its  mgurnems  with  a  new  argument  frcmi  the  followtq).  IntetjMeting  dlqiticals  in  terms  of  the  previ¬ 
ous  sratoice  of  tiie  currait  discourse  segnmnt  is  widely  accqtted  [  14],  but  how  it  is  drnie  in  any  given  NLP 
system  dqtends  on  tiie  system’s  particular  rqnesent^kHial  formalisms.  The  Eucalyptus  fnek^cal  form 
was  a  suiuMe  rqxesentatkm  in  two  ways:  the  case  frame  incorporates  the  semantic  type  constraints  of  its 
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arguments,  and  the  form  also  retains  some  syntactic  information,  in  paiticular  identifying  the  syntactic  sub¬ 
ject 

Eucalyptus  handles  three  basic  classes  of  elliptical  sentences.  The  first  consists  of  simfde  repetitkm 
fonnulae  like  Again,  which  simply  cause  the  prior  utterance’s  prelogical  form  to  be  re-evalua^: 

Increase  the  simulation  speed. 

Again.  [Increases  it  a  second  time] 

The  second,  VP-ga{^)ed  sentences,  are  inteq>reted  by  making  a  copy  of  the  prior  utterance’s  prel¬ 
ogical  form  to  serve  as  tlK  eUded  clause  and  replacing  its  subject  operaiKl  by  the  subject  of  the  elliptical: 

Which  F14s  are  moving  to  station  1  ? 

Which  UAVs  are?  [Which  UAVs  are  moving  to  station  1?] 

Is  fighter  I  moving  to  station  I  ? 

Is  fighter  27  [Is  filter  2  moving  to  station  1?] 

Is  fighter  1  moving  to  station  I  ? 

Recall  a  UAV  that  is.  [Recall  a  UAV  that  is  moving  to  station  1.] 

In  the  third  class,  the  subject  of  the  elliptical  replaces  one  of  the  arguments  (not  necessarily  the 
subject)  of  the  prior  utterance’s  logical  form.  Eucalyptus  searches  the  prelogical  form  of  the  prior  utterance 
for  an  argument  slot  that  will  semantically  accept  the  subject  of  the  elliptical,  and  performs  the  subsdtu- 
ti<ni: 

Are  any  F14s  moving  to  station  1? 

Any  UAVf}  (Are  any  UAVs  moving  to  station  1?] 

What  about  station  5?  [Are  any  UAVs  moving  to  station  5?] 

I  meant  station  4.  [Are  any  UAVs  moving  to  statitm  4?] 

Move  fighter  I  to  station  5. 

Fighter  2  also.  [Move  fighter  2  to  station  5.] 

Make  that  station  4.  [Move  fighter  2  to  station  4.] 

Elliptical  inputs  of  this  type  can  also  be  used  to  correct  speech  recognition  errors:* 

Is  fighter  4  moving  to  the  tanker?  [System  heard  “fighter  14”,  “that  tanker”] 

I  said  fighter  4.  [Move  fighter  4  to  that  tanker.] 

The  tanker.  [Move  fighter  4  to  the  tanker.] 

Embedded  clauses  are  included  in  the  search  for  the  argument  to  replace.  If  more  than  one  substitution  is 
possible  at  the  same  clause  level,  the  system  reports  the  ambiguity  to  the  user.  When  two  possible  substitu¬ 
tions  occur  at  different  clause  levels,  however.  Eucalyptus  does  not  report  an  error  but  just  replaces  the 
highest  level  one,  as  in  the  final  example: 

Which  fighters  moving  to  station  5  are  low  on  fuel? 

What  ^ut  station  4?  [Which  fighters  moving  to  station  4  are  low  on  fuel?] 

Have  fighter  1  relieve  fighter  2. 

Same  with  fighter  3.  [System:  “Sorry,  don’t  understand.”] 

Is  an  F14  moving  to  a  tanker  that  is  r^ueling  a  UAV? 

What  about  an  EA6B?  [Is  an  EA6B  moving  to  a  tanker  that  is  refueling  a  UAV?] 

This  last  rule  is  really  a  preference  heuristic  (i.e.,  “Is  an  F14  moving  to  a  tanker  diat  is  refueling  an 
EA6B?”  is  a  valid  but  less  likely  interpretation),  and  so  should  probably  be  reported  as  an  ambiguity  until 

*Tliiis  comcled  NPt  are  not  removed  from  the  diacome  history  but  are  **ina8lced”  by  the  more  recent  cmrecting  NPs. 
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Eucalyptus  is  {Movided  with  more  ftiUy  developed  NL  output  capability  to  query  the  user  for  the  intended 
meaning. 

MULTIMEDU  INTEGRATION 

A  i»imaiy  c^jective  in  Eucalyptus  is  not  to  doncmstrate  diat  NL  is  a  superior  interface  medium  to 
grtqrhics  or  vice  versa,  but  that  the  two  have  fiindamentally  different  and  comidementaiy  strengths  diat 
rfumirf  both  be  availaUe  to  the  user  simultaneously.  This  secticm  explores  integral,  mixed-media  transac- 
tkms  in  udiich  graphics  and  NL  cmnbiiK  and  interact 

NL  Input  to  Dialogue  Windows 

As  we’ve  seen,  the  KOALAS  GUI  uses  popup  dialogue  windows  to  pnxnpt  the  user  for  arguments 
and  parameters  needed  to  comidete  a  proposed  fighter  order  or  threat  assessment  operatkm.  Each  window 
contidns  (me  or  more  labelled  text  fiel^  (for  tumeric  information  such  as  ID,  bearing,  arrd  course)  and  may 
also  (xmtain  (me  or  two  labelled  cyde  items  for  choosirtg  fnm  dosed  sets  of  parameter  values  su<di  as 
speed  (Sow,  Fast,  Statkmary)  and  aircraft  type.  These  windows  do  not  just  passively  accq;)t  user  irqut  to 
their  slots,  tut  have  a  high-levd  irgwt  bdiavior  that  prompts  for  certain  argumoits  in  a  particular  order. 
When  the  window  r^rpears,  die  cursor  is  positkmed  in  the  first  text  fidd  and  the  window  dismays  a  message 
{xompting  the  operator  to  select  an  object  or  positkm  on  die  radar  screen  with  the  mouse.  Orice  sjbe  (kies 
so,  the  system  auttmiatically  fills  in  one  (»-  more  text  fields  in  the  window  (sometimes  setting  a  choice  in  a 
cycle  item  as  well),  die  cursor  moves  to  the  next  empty  text  fidd,  and  a  new  prompt  is  dis|dayed.  When  no 
empty  fields  remain,  the  syston  pnmipts  die  operator  to  dick  once  more  ftir  final  ctmfirmation.  The  ri^ 
mouse  butttm  can  be  used  as  an  “undo”  (^rstor  at  any  time,  retracting  the  jmevious  moused  data  entry 
operatkm  by  clearing  the  relevant  text  fields,  res^ting  the  cursor,  and  dis{daying  the  previous  prompt 
Although  the  user  can  also  interact  directly  with  the  cyde  items  or  text  fields  using  die  mouse  and  key¬ 
board,  this  itqwt  mode  is  not  interactive  with  the  hi^-level  irqwt  mode,  so  die  user  cannot  type  certain 
fields  and  then  use  die  mouse  to  fill  odiers.  However,  the  keyboard  can  be  used  toedit  (vtdiange  values  diat 
haive  already  been  entered  by  mouse  irqwt 

Alduu^  the  low-level  operatkm  of  manually  typing  data  into  a  labdled  text  field  ctmstitutes  a 
“dialogue”  techni(]ue  fttmi  die  staiKhmint  of  basic  human-computer  interaction,  the  higher  level  iiqwt 
mode  oi  the  KOALAS  dialogue  windows  more  closely  resemUes  a  true  NL  ditdogue:  die  window  prompts 
die  user  ftnr  inftirmatitm  with  a  soitmice  like  “Click  Left  Buttcm  on  Hghter”  and  the  user  reqxmds  widi  a 
referendal  gesture  by  grqdiicaUy  selecting  an  object  or  location  (m  die  radar  screen.  While  Ei^yptus  was 
originally  intended  to  <^y  accept  elliptical  irqiuts  as  followups,  given  the  dialogue-like  nature  of  the 
KOALAS  popup  window  interactions,  it  was  decided  that  die  user  riwuld  be  allowed  die  (^(m  of  qieak- 
ir^  ringular  referring  exfnessions  (proper  names  and  noun  [dirases)  in  response  to  die  wi^w  prompts, 
analogDus  to  die  (dicks  on  die  radar  screen  (grqdiical  referaices)  already  supported  by  the  GUI. 

Each  refOTing  expresskm  directed  to  a  dialogire  window  undergoes  parsing,  semantic  interineta- 
tkm  and  (kmudn-model  dierefermicing  just  like  those  contained  in  sentential  inputs  to  the  systan.  If  the 
resultitig  sin^eTOn  referent  is  of  the  proper  semantic  type  to  fill  the  cutrem  text  field,  its  ID  number  is 
odered  into  the  fidd,  die  cursor  is  advam^  to  the  next  anpty  slot,  and  die  system  pnmipt  iqidated,  exacdy 
paralleling  the  system’s  feedback  to  grtqdiical  input  The  entered  data  respo^  to  die  “undo”  (right-click) 
grrqdiical  operator  just  die  same  as  grrqihically  entered  data.  The  resulting  interactitm  is  analogous  to  a  nat¬ 
ural  lai^uage  dialogue  su(di  as  the  following: 

User  <Mk>ve> 

System:  Select  a  fighter. 
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User  One  of  the  F14s  holtUng  station  2. 

Syston:  That’s  fighter  1.  Select  a  statioiL 

User:  The  same  station  fighter  11  is  moving  to. 

Syston:  That’s  station  S.  Okay? 

User  Okay. 

Eucalyptus  does  not  pennit  the  user  to  select  and  speak  to  a  text  field  other  than  die  one  currently 
being  {Mcmpt^  for.  since  that  would  be  equivaloit  to  bypassing  the  window’s  hi^-level  ii^t  behavior  by 
using  the  Ireyboard  for  raw  numeric  data  entry.  TMs  is  omsistent  with  the  initial  working  decision  dutt 
Eucalyptus  be  a  qxrken  NL  system,  not  merely  a  voice  input  system;  the  role  of  speech  irqwt  in  Eucalyptus 
is  not  to  serve  as  a  ”vocal  keyboard”  for  entering  raw  dr^  but  to  take  part  in  a  higher-level  NL  dialogue 
about  concqttual  enddes  (i.e..  NL  references),  vidiich  is  why  Eucalyptus  treats  NPs  as  semantically  equiva- 
leitt  to  the  GUI’s  high-level  gnqphical  denotadon  of  objects. 

As  mentioned  earlier,  Eucalyptus  permits  correctkms,  queries,  advice  interactions,  and  system 
control  commands  as  “asides”  during  dialogue  window  data  entry.  SiiKe  the  system  assumes  that  isolated 
noun  phrases  ate  subdialogue  interactkms  with  the  window,  however,  nich  asides  cannot  include  ellipdcal 
followups  consisting  of  bare  noun  phrases.  This  is  a  case  where  a  discourse  component  oqMdde  of  manag¬ 
ing  more  dun  a  single  context  ^Mce  [IS]  mi^  be  useful:  if  the  user  could  verbify  cue  the  return  from  an 
aside  to  die  window,  dds  lestricdon  against  using  bare  NPs  would  no  longer  apjdy. 

Dialogue  Wfaulows  for  NL  Command  CompIctkNi 

As  seen  earlier,  verbal  fighter  ooittrol  and  threat  managemetd  commands  always  bring  up  the  same 
dialogue  windows  used  for  data  entry  in  die  graphical  iraerfoce,  but  widi  the  verbalized  arguments  already 
filled  in.  This  saves  four  purposes:  to  provide  visual  axifimudion  that  the  system  understood  and  imer- 
preied  the  input  omrecdy;  to  allow  die  usa  to  alter  diose  arguments  that  have  been  entered  so  fiu;  to 
prompt  the  user  to  fill  in  any  remaining  umqiecified  infonnadort.  and  to  give  die  user  firud  accepthejea 
oppwtuniQr.  These  are  the  same  four  roles  diat  die  dialogue  window  (days  in  die  origirud  graphical  inia- 
so  it  was  felt  dut  using  the  s«ne  feedback  device  for  NL  irqwt  would  be  an  integrated  and  consistent 
use  of  system  resources  that  would  avoid  complicating  die  intofsce  widi  additional  output  modalities  like 
NL  (e.g.,  “Havir^  fighter  2  relieve  fighter  12  to  refuel  at  tardea  1,  okxyT*),  and  that  providing  the  same 
unamtdguous,  cttutrical  feedbadt  tedmique  to  die  usa  in  all  cases  mi^  be  less  confusing  and  demanding 
on  die  usa’s  attemkxi  (aldiough  ahetha  it  actually  does  in  fact  reduce  the  usa’s  cognitive  load  would  of 
course  terpiire  expoimental  testing). 

One  feature  of  die  KOALAS  grafdiical  interface  is  to  pnddbit  the  usa  fenn  issuing  anodia  Hghta 
Managemad,  Threat  Assessment,  or  Eiqierimenta  Qxitiol  operation  while  a  dialogue  window  is  still  on 
die  screen,  beqdng  and  flashing  die  window  to  signal  that  it  must  be  comideted  and  dismissed  first  Euca¬ 
lyptus  parallels  this  behavior  by  beephig  md  flashing  die  window  if  die  user  issues  a  NL  command  of 
diose  s(»ts  addle  a  diafogue  wiridow  is  still  pending.  Any  odia  command  type  or  query  is  acoqitabie,  as  is 
also  true  in  the  grtqddcal  interface  (e.g..  die  user  can  interact  with  die  cunoit  advice  ot  scroll  through  the 
(flqdiqr  windows  during  a  dialogue  window  irderaction). 

Dtoeowseffistory  for  Gr^ihiad  Opcratioris 

Wheneva  a  domain-functional  operation  is  performed  using  die  KOALAS  gnqddcal  imerfece, 
Eucahrptus  generates  an  equivalent  prelog^  form  similar  to  diose  that  result  from  verbal  irgiut  and  enters 
Disooim  Entities  finr  each  of  the  operation’s  imididt  or  explidt  arguments  into  the  discourse  contact  This 
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allows  two  kinds  of  interactions  between  graphics  and  NL.  First,  the  grai^cal  operation  and  its  arguments 
become  availaUe  as  coraext  for  interpreting  an  elliptical  follow-up  command: 

^httr  ixumker  I> 

Same  withfighur  5.  [=  refuel  fighter  S  at  tanker  1] 

Second,  its  aiguments  become  available  as  antecedents  for  anaphoric  reference: 

<Mov€>  ^fighter  14>  estation  1> 

What  bearing  does  he  have?  [/re  =  fighter  14] 

In  addition  to  Hghter  Management  operations.  Threat  Assesanent,  Experimenter  Control,  and  Advice 
IMndow  operations  also  generate  context  history,  allowing  irtteractions  such  as  the  following: 

Khiake  Threat> 

Vector  fighter  1  there.  [there  =  to  new  threat] 

Advieo 

Accept  it.  [it  =  current  advice] 

<Shaw  Real  Threatx> 

Dim  diem  off-  [them  =  real  threats] 

Deictic  Rcforoioe 

Deixis  is  a  form  of  reference  in  which  pointing  gestures  accompany  NL  referring  expressions.  It  is 
a  powerful  and  useful  mechanism  in  human  communication  because  each  component  of  the  reference  can 
be  minimally  ^tecific,  but  in  omnlnnation  can  constrain  or  disamlxguate  each  other  sufficiently  to  yield  an 
unambiguous  denotation. 

A  deictic  reference  in  Eucalyptus  consists  either  of  the  word  here  or  a  Ox)ssibly  headless)  NP  hav¬ 
ing  the  dononstrative  this  or  these  as  determiner  (e.g.,  this,  these,  these  hypothetical  Badgers,  this  fighter 
vectoring  to  a  threat),  accompanied  by  one  or  more  middle  mouse  clicks  on  the  radar  screen.  As  each 
moose  event  occurs,  its  x-y  coordiiutes  are  ^red  in  a  list  for  Uttn*  access  by  the  NLP  cmnponott.  During 
NL  processing,  these  mouse  events  are  correlated  sequoitially  with  dw  deictic  NPs  in  die  sentence.  The 
referent  for  each  NP  is  then  determined  by  finding  the  subs^  of  KOALAS  entities  located  at  or  nearest  to 
die  moose  event’s  x-y  coordiiutes  dut  conforms  to  die  NP’s  semantics  Qnduding  the  semantic  prestqiposi- 
tkms  of  its  host  predfeate).  “Nearest  to’’  is  defined  as  the  closest  aircraft  within  a  distance  of  10  inxels  from 
die  moose  coonfinates,  which  is  how  it  is  defined  for  direct  reference  in  the  original  grs^cal  interface. 

For  exanide,  a  click  on  a  tanker  aircraft  could  be  a  reference  either  to  the  aircraft  itself,  to  the  radar 
screen  sector  (duty  station)  at  those  coordinates,  or  to  die  positkm  indicated  by  the  coordiiutes  dwmselves. 
Accompanied  NPs  like  this  aircrefi,  this  friendly  aircrtffi,  this  support  aircrtfft,  this  Uinker,  or  this  E2C, 
die  aircraft  is  chosen  as  referent.  Sintilaily,  this  sector  or  Ms  station  selects  the  duty  station  and  this  posi¬ 
tion  selects  the  screen  coordinates.  For  an  NP  like  this  enemy  aircraft,  this  fighter,  or  dUs  F14,  however,  the 
imeiaection  is  empty  and  Eucalyptus  responds  “No  such  (aircraft,  fighter,  F14)  exists.’’ 

The  mqiression  here  is  maximally  vague  and  in  the  same  examide  coukl  be  a  reference  either  to  the 
taidcer,  die  screen  sector,  or  the  coixdiiutes.  In  dut  case.  Eucalyptus  uses  die  semantic  presuppositkais  of 
die  eiqiression’s  host  predicate  to  try  to  resolve  the  reference.  For  examfde,  in  the  cmitext  of  the  imperative 
aoemiee  Move  fighter  1  here,  the  verb’s  predicate  p-move-TO-STATION  expects  a  duty  station  as  its 
Mgnmwit  and  so  diooses  die  sectOT  as  refeent.  hi  a  query,  however,  move  can  mean  either  p-move-to- 
STATION  or  P-MOVE-TO'TANKER,  SO  Is  fighter  I  moving  here?  receives  both  meanings:  the  qrstem 
answers  “Yes”  if  flatter  1  is  moving  either  to  that  staticm  or  to  the  tanker.  In  the  context  of  Refuel  fighter  I 
km,  the  preificaie  P-REFUEL’s  sonaittic  eiqiectatkms  cause  just  the  tanker  to  be  dwsen  as  referent,  and 
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with  the  input  Create  a  hypothetical  Badger  here,  the  system  chooses  the  radar  screen  coordinates.  Finally, 
given  an  utterance  like  Vector  fighter  1  here  (where  P-VECTOR  presupposes  an  enemy  target  as  argu¬ 
ment),  the  intersectitm  set  is  empty  and  the  reference  component  re^xHids  “No  such  threat  exists.”  Similar 
sorts  of  presuppositional  behavior  can  be  observed  by  clicking  on  the  radar  screen  and  asking  questions 
sudi  as  Who  is  this.  What  is  UUs,  What  aircrcfi  is  this,  etc. 

The  (teictic  reference  mechanism  used  in  Eucalyptus  is  very  similar  to  the  direct-reference  media- 
nism  for  moused  object  denotation  in  the  KOALAS  graidiical  interface,  where  the  dialogue  window  slot 
being  prmnpted  for  expects  an  argument  of  a  certain  sort  (friendly,  fighter,  threat,  station,  or  position)  and 
the  interface  selects  the  object  of  that  sort  closest  to  the  mouse  coordinates.  Thus,  if  a  fighter,  tanker,  and 
hypodietical  direat  were  in  such  close  proximity  that  their  icons  overlapped  on  the  radar  screen,  the 
KOALAS  grai^cal  interface  could  still  select  the  object  satisfying  the  type  expectation,  just  as  Eucalyptus 
can  using  deictic  reference  (e.g.,  this  F14,  this  EA6,  this  Badger).  In  the  more  conventional  direct-manipu¬ 
lation  tqtproach  where  each  icon  would  be  associated  with  a  mouse-sensitive  active  region,  only  die  tqiper- 
most  of  the  overltq^nng  icons  would  be  visible  m  the  mouse  and  could  be  selected,  requiring  additional 
etqiose/hide  (^rations  to  access  the  hidden  objects. 

IVvo  siffl|difying  assumptions  are  made  in  Eucalyptus's  treatment  of  deixis.  First,  only  one  of  the 
deictic  exiMessions  in  a  sentence  may  be  plural.  Second,  the  restrictive  choice  of  deictic  vocabulary  (this, 
diese,  and  here)  ensures  that  no  ambiguity  exists  between  deicdcs  and  discourse  anaidiors,  which  must  use 
the  distinct  set  of  demraistratives  duu,  those,  and  there.  In  general  English  this  distinction  is  not  made  ~  for 
example,  die  phrase  “this  distinction”  was  just  used  anaphorically,  and  one  can  refer  deictically  to  this  book 
and  that  one  to  indicate  proximity  to  the  speaker.  Since  deictic  references  in  Eucalyptus  all  involve  figura¬ 
tively  “touching”  referenced  entities  by  clicking  on  them  with  the  mouse,  however,  tiie  strictly  proximate 
terms  are  more  aptnopiiate,*  and  a  user  interaction  study  to  be  discussed  later  suggests  that  KOALAS 
users  may  tend  to  verbalize  deictics  and  anaphors  using  these  same  vocabulary  distinctitHis. 

These  two  craistraints  allow  tiie  system  to  perform  a  simple  left-to-right  assignment  of  mouse 
events  to  NFS  witinut  requiring  any  time  correlations.  Fbr  examine,  a  smitence  with  multiple  plural  deixis 
like  Have  these  F14s  and  diese  UAVs  refuel  would  require  time  correlations  to  assign  a  partition  of  the 
mouse  evoits  to  each  deictic  expressiort  Similarly,  if  a  single  referring  expression  could  be  either  ana¬ 
phoric  or  deictic,  then  a  sentence  like  Have  that  tanker  refuel  that  fighter  accompanied  by  a  single  mouse 
dick  would  require  time  correlation  to  determine  which  NP  should  be  interpret^  as  an^^ric  (i.e.,  that 
tanker  or  fighter  mentioned  earlier)  and  which  as  deictic. 

The  use  of  timestamps  is  avoided  because  while  it  is  easy  to  timestamp  mouse  evoits  as  they 
occur,  timestamping  deictic  NPs  in  Eucalyptus  is  more  problematic.  This  is  because  the  initial  phonetic 
prooesang  takes  place  in  hardware  indeperident  of  the  host  computer,  so  altiiough  the  transcribed  speedi 
data  does  indude  the  overall  duration  of  the  uttertmce  as  well  as  the  relative  time  position  of  each  word  in 
it,  the  syston  cannot  know  what  time  the  utterance  actuaUy  began.  Since  it  does  know  the  arrival  time  of 
the  decoded  signal  at  the  host  computer,  however,  if  it  were  to  assume  a  small  fairly  constant  or  linear 
decoding  time  then  it  would  be  possible  to  extrrqxrlate  with  reasoruible  accuracy  the  time  at  whidi  eadi 
ddctic  NP  began  (that  is,  whoi  each  word  this,  diese,  or  here  occurred).  Eucalyptus  does  in  fact  timestamp 
ddctic  NPs  in  just  this  way,  in  the  event  tiiey  become  needed  fw  future  extensions  of  the  system. 

hi  the  originai  grrqrhical  interface,  tiie  left  mouse  button  is  used  to  sdect  the  operands  to  fighter 
orders  and  tiueat  managmnait  commands,  so  ideally  the  same  button  diould  have  been  u^  for  ddctic  ref¬ 
erence  as  wdl.  But  since  in  Eucalyptus  the  host  computer  does  not  know  when  qieech  irqxit  is  occurring 
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until  after  the  press-to-talk  switch  has  been  released  at  the  end  of  the  utterance,  the  system  would  not  be 
able  to  distinguish  deictic  from  ordinary  grt^Mcal  input  if  the  same  tnitton  were  used  for  both.  Besides, 
when  no  dialogue  window  is  pending,  the  left  mouse  ^es  on  a  secmidary  functionality  (bringing  up  a  sta¬ 
tus  information  window  for  an  aircraft)  that  also  could  not  be  distinguished  from  deixis.  The  other  two 
mouse  buttons  already  have  functional  definitions  in  the  original  interface  and  so  are  unavailable  for  deic¬ 
tic  reference:  middle  click  centers  the  radar  display  at  the  mouse  coordinates,  and  right  click  displays  the 
threat  sittings  for  a  UAV  or  hypothetical  threat  Adding  the  Control  key  to  left-click  for  deictic  reference 
would  have  been  awkward  since  the  user’s  non-mouse  hand  should  be  left  free  to  control  the  press-to-talk 
switch.  Therefore,  it  was  decided  to  reassign  the  original  mouse-middle  functionality  (radar  screen 
recenter)  to  Control-mouse-middle,  freeing  up  mouse-middle  for  deictic  reference. 

An  alternative  would  have  been  to  require  no  mouse  click  at  all  during  deictic  reference,  but  to  just 
take  die  x-y  coordinates  of  the  mouse  at  the  time  the  deictic  NP  was  uttered.  Since  the  NP’s  timestamp  can¬ 
not  be  extrapolated  and  assigned  imtil  after  the  complete  utterance  has  been  decoded,  however,  it  would 
have  been  necessary  to  keep  an  ongoing  history  of  mouse  locations  at  all  times  during  the  simulation  for 
possible  correlation  later  with  deictic  NPs.  If  demonstrative  NPs  were  allowed  to  be  either  deictic  or  ana- 
[dioric  (as  in  the  example  Have  that  tanker  refuel  that  fighter  discussed  earlier),  die  problem  of  correladng 
possible  deictic  references  with  mouse  coordinates  would  be  compounded;  the  mouse  just  might  happen  to 
be  positioned  over  a  fighter  icon  when  the  speaker  uttered  what  was  intended  to  be  an  an^dioric  reference, 
resulting  in  a  misinterpretation.  Requiring  that  mouse  clicks  accompany  deictic  references  avoided  these 
problems. 

When  the  left  mouse  button  is  used  to  graphically  denote  an  aigument  to  a  dialogue  window,  the 
user  gets  immediate  visual  feedback  of  die  success  of  die  operation  -  the  argument’s  ID  appears  in  die  cur¬ 
rent  window  slot.  However,  the  user  does  not  find  out  whether  a  deictic  reference  was  successful  until  the 
accompanying  speech  input  has  been  fully  processed  and  the  dialogue  window  is  displayed  (or  in  the  case 
of  a  question,  until  the  query  answering  facility  responds).  To  reassure  the  user  that  the  deictic  mouse  elide 
was  effective.  Eucalyptus  provides  visual  feedback  by  blinking  the  icons  of  all  the  potential  referents  at  or 
near  the  mouse  coordinates  when  the  click  occurs. 

Threat  Reference 

The  ability  to  view  the  actual  threats  during  a  KOALAS  simulation  is  treated  as  a  spedal  “God’s 
eye  view*’  that  is  normally  not  invoked,  since  the  KOALAS  fdiilosoidiy  is  that  the  operator  only  create  and 
interact  widi  hypodieses  about  what  threats  might  really  be  out  there.  As  a  result,  the  Threat  Status  window 
di^lays  status  information  only  for  hypothetical  thre^  unless  the  Real  Threats  switch  is  turned  on,  in 
whidi  case  real  threats  receive  status  entries  in  the  witKlow  and  their  blips  are  added  to  the  radar  screen  dis- 
fday  (at  which  point  they  can  be  referenced  grtqjhically,  i.e.,  moused  on).  Since  one  Eucalyptus  design 
obj^ve  was  for  the  NL  and  graf^cal  interfaces  to  have  exactly  parallel  semantic  behavior.  Eucalyptus 
does  not  make  real  threats  availt^le  for  NL  referertce  unless  they  are  graphically  visible.  For  example. 
What  is  threat  I’s  bearing?  asked  with  the  Real  Threats  switch  off  accesses  hypotiietical  threat  1  (if  sudi 
exists),  otherwise  respmiding  “No  such  threat  exists”  even  if  a  real  threat  with  that  ID  exists.  With  the 
switch  on,  if  bodi  a  real  and  hypothetical  threat  1  exist,  the  system  responds  “More  than  one  such  threat 
exists,”  but  otherwise  reports  on  the  one  that  does.  Under  the  same  circumstances,  however,  die  irqrut  Vec¬ 
tor  fighter  14  to  threat  1  (where  vector  semantically  presuf^ses  a  hypothetical  threat  as  argument,  sitKe 
(me  cannot  order  fi^liters  to  vector  to  real  threats)  does  ix)t  result  in  a  refemtee  ambiguity.  Finally,  the  user 
can  always  qu^fy  the  threat  reference  with  the  appropriate  adjective  {real  threat  1,  the  hypotiietical 
threats)  if  more  specificity  is  needed. 
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EUCALYPTUS  ARCHITECTURE 

Figure  2  shows  the  architecture  of  Eucalyptus.  The  three  main  modules  of  the  system  are  (counter- 
dodcwise  from  upper  left)  the  speech  recognition  component,  natural  language  processor,  and  iq)plication- 
specific  translator,  interfacing  to  the  taiget  system  in  the  upper  right  These  cmnponrats  are  discussed  in 
more  detail  in  the  sectitms  that  follow. 

Speech  Recognition  Componoit 

Eucalyptus  uses  the  Speech  Systems  Inc.  (SSI)  Phonetic  Engine  200  continuous  q)eedi  recognizer. 
The  SSI  system  requires  a  finite  state  grammar  of  all  acceptable  input  utterances  (compil^  fnmi  a  data  file 
written  in  context-firee  iwtation)  which  it  uses  to  find  the  closest  matdi  to  the  i^ion^e  sequoice  it  has 
derived  from  the  iiqmt  speech  signal.  The  speech  grammar  for  Eucalyptus  consists  of  about  250  ncmitera- 
tive,  nonrecursive  {mxluctions  capable  of  recognizing  about  1 15  million  different  ^ken  utterances.  Since 
the  rqnesentational  formalism  ody  supports  context-free  notation,  syntactic  and  sonantic  constraints  are 
encode  in  the  grammar  by  proliferating  nonterminal  symbols,  resulting  in  a  so-called  semantic  grammar 
as  tqjposed  to  a  i^irase  structure  grammar. 

The  subset  {RKUietic  dictionary  for  this  application  contains  just  under  260  words.  Twenty-five  of 
diese  are  integers,  seven  are  alphabetic  characters  (fm*  speaking  acronyms  like  UAV  and  EA6B),  34  are 
qrostrqMzed  {Rurals  (for  spealdng  genitives  like  tite  fighter’s  said  fighter  I’s),  and  die  mnainder  are  ordi¬ 
nary  Eriglidi  words,  cif  these,  eight  (badgers,  koalas,  pathfinder,  padifinders,  phoenixes,  rqwsidon,  vec¬ 
tored,  and  vectoring)  were  not  present  in  ttie  SSI  standard  phonetic  dictitmary  and  had  to  be  added  using 
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their  dictkmaiy  i^idate  facility.  Eucalyptus  allows  the  user  to  pronounce  aircraft  ID  numbers  either  as  sin¬ 
gle  words  {nineteen,  twenty)  or  as  their  individual  digits  {one  nine,  one  tuner,  two  zero,  two  oh),  the  latter 
often  providing  improved  recognitkm  accuracy. 

Although  recursive  ptrase  structures  such  as  OMnpounds,  nested  prqwsititmal  (dirases,  and  rela¬ 
tive  dauses  present  no  problem  to  the  natural  language  i»ocessing  (NLP)  comptment  described  in  the  next 
section*  add^  sudi  structures  to  die  speech  recognitkm  grammar  increases  its  size  and  reduces  its  recog¬ 
nition  accuracy  to  such  a  degree  that  it  is  not  {uofitaUe  to  do  so.  In  particular,  attempting  to  add  relative 
dauses  to  the  qieedi  grammar  actually  renders  it  too  large  to  compile.  This  is  unfortunate  since  reladviza- 
tkm  is  (me  of  the  most  powerful  linguistic  mechanisms  for  constructing  the  descriptive  doiotational 
ejqiressions  that  make  nat^  language  such  a  desirable  adjunct  to  gn^hical  interfacing.  At  present,  dien, 
sentences  that  use  recursion  must  be  entered  by  keyboard  rather  than  ^koL 

The  SSI  system  provides  genetic  (speaker-indepoident)  male  and  female  speaker  models  that 
worit  (juite  well  within  die  constrained  syntax  and  semantics  of  the  Eucalyptus  domain.  The  most  recogni- 
tkm  errors  are  encountered  in  two  context-insensidve  word  dasses:  determiners  {a,  the,  this,  that)  and  ID 
numbers  {thirteen,  fourteen).  Very  short  sentences  (Dkoy,  Cancel  that,  Threat  2,  Exit)  tend  to  yield  the 
most  serious  misrecognitions  since  they  provide  the  least  amount  of  phonetic  context  for  the  speech 
decoder  to  idendfy  die  intended  predicate. 

NatundlMuguage  yfindow 

Eucalyptus  adds  erne  window,  the  Natural  Language  Window  (Fig.  3),  to  the  KOALAS  tool’s 
graphical  display.  The  ASOI  transcriptiem  of  the  spokoi  utterarux  is  echoed  in  a  text  fiedd  and  then  passed 
automatically  to  the  rest  of  the  system  few  NL  pHOcessirig  and  execution  unless  die  Confirm  Speech  check 
switch  is  turned  cm,  in  which  case  the  Parse  tetton  must  be  used  to  continue  pmxxssing.  This  allows  the 
user  the  cqipottunity  to  dieck  the  accuracy  of  the  text  and  perhaps  make  minor  edits  to  it  with  mouse  and 
keyboard  before  soiding  it  on  for  ftirdier  i»ocessing.  With  Confirm  Speech  off,  die  system  can  be  oper¬ 
ate  ocHDidetely  hands-off  using  vcnce  only.  Altemadvdy,  the  user  can  clear  die  text  field  with  the  Clear 
butttm  and  enter  an  entire  soitmce  via  the  keyboard  followed  by  jnessing  the  Parse  buttoa  Finally,  bdow 
die  text  ir^ittt  field  is  a  write-only  text  ou^iut  field  where  Eucalyptus  {Mogress  messages  and  final  responses 
qipear. 

Natural  Language  Processor 

After  a  sentence  string  has  been  transcribed  by  the  speech  component  (or  typed  d  die  keyboard),  it 
passes  to  the  natural  language  understanding  compement  for  linguistic  analysis  and  conversion  to  logical 
form.  The  NLP  compemem  cemsists  of  fimr  modules;  the  PROTEUS  parser  developed  at  New  York  Uni- 
verdty’s  Gourant  fostitute  [16],  and  the  TINSEL  case-frame  interpreter,  FOCAL  reference  resolution  mod¬ 
ule,  1^  FUNTRAN  quandfi^-expresaon  builder,  all  diree  developed  in-house.  The  Appoidix  illustrates 
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die  varkMis  stages  of  iMocessing  for  a  tyjMcal  input  soitence,  disaissed  in  more  detail  in  the  subsec¬ 
tions  that  follow. 

Symtaetie  Amafysis 

The  mOTEUS  syntactic  grammar  used  in  &icalyptus  is  based  on  a  streamlined  NL  interface 
grammar  developed  for  the  InteiFIS  {xoject  [17].  and  cmisists  of  approximately  180  context-foee  phrase 
structure  rules  and  50  procedural  syntactic  restriction  rules.  Besides  covering  the  sami*  ’’'put  data  as  die 
^leech  recognition  grammar,  it  also  covers  con^xiunding  (on  imperatives,  questions,  a  u  noun  {dirases), 
iteration  of  NP  modifiers,  and  recursive  embedding  within  NP  posunodifiers  (relative  clauses  and  fwqxisi- 
tional  [dirases).  The  PROTEUS  lexicon  contains  about  425  words,  nearly  twice  as  large  as  the  spe^  rec¬ 
ognition  lexicon.  Most  of  the  extra  words  are  unused  morphological  variants  (verb  aspects  and  noun 
idurals)  generated  automatically  from  root  forms  by  die  PROTEUS  lexical  macros. 

The  parser’s  input  is  the  ASQI  string  contained  in  the  Natural  Language  Window  text  input  field, 
either  typed  from  the  keyboard  or  transcribed  from  speedi  (Appendix,  example  1).  The  parser’s  ouqiut  is  a 
regularized  form  suitable  for  smnantic  interpretation  called  the  Intermediate  Syntactic  Rquesentation 
(ISR)  [18].  Among  the  regularizations  that  occur  are  verb  phrase  depassivization,  ttdcenization  of  idimns 
and  stock  fdirases  (auto  accept,  as  well),  translation  of  numeric  phra^  (twenty,  three  quarters)  into  num¬ 
bers  (20,  3/4),  reduction  of  multi-tdcen  pronunciations  (one  niner,  e  two  c)  into  individual  tcdcens  (19, 
E2C),  and  extractitm  of  the  negation  curator  NOT  from  verb  contractions  (aren’t,  didn’t). 

Sememde  Intarpretation 

The  predicate-argument  interpreter  TINSEL  [19]  runs  in  tandem  widi  the  parser,  pattern-matching 
each  ISR  against  a  hierarchy  of  about  160  semantic  frames  (typed  slot-filler  structures)  and  a  rou^y  equal 
number  of  syntactic-semantic  miqiping  patterns,  inserting  semantic  class  and  thematic  role  information 
into  the  ISR  (Appendix,  exam[de  2)  to  yield  a  semantically-marked  version  (Appendix,  examfde  3).  Dur¬ 
ing  intmi»etati(m.  TINSEL  sq^es  pre^cate  domain  constraints  (selection  restrictimis)  to  prune  anoma¬ 
lous  interpretations  and  thus  hdp  guide  the  parse.  TINSEL  performs  five  primary  interiHetive  functitms: 

•  Many-to-<me  mapping  of  syntactic  to  semantic  roles  (thematic  resolution).  For  examine,  in  each 
sentoice  of  (29),  j^hter  2  msqrs  it?  tiie  :  Instrument  role  even  though  it  is  the  sub^  of  the  first 
and  prqxrsitioiudly  marked  wUh  M  fhe  secoid. 

(29)  Fighter  2,  reUeve  fighter  I. 

Relieve  fighter  1  with  figlder  2. 

•  Many-ttHHie  mrqqnng  of  lexical  to  semantic  classes  (synonymy),  for  example  meppiag  die  verb 
in  each  sentence  of  (30)  to  the  predicate  P-RECALL. 

(30)  Fighter  1,  return  to  the  carrier. 

Recall  fighter  1. 

Have  fighter  I  come  back  to  the  carrier. 

•  One-to-many  mapffing  of  syntactic  to  semantic  txdes  (thonatic  amlnguity).  For  examfde,  in  the 
first  sentence  of  (31),  the  ssAypctthataircrefimsps  to  the  :patlent  role  (the  erttity  affected  by 
tile  verb),  D^reas  in  the  second  it  mtqs  to  tiie  :  instrument. 

(31)  Is  dutt  aircraft  rqfiteling? 

Is  dutt  aircreft  refueling  anyone? 
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•  One-to-many  mapping  of  lexical  to  semantic  classes  (polysemy),  for  example  mapping  the  noun 
fiul  {How  much  fuel  does  fighter  1  hovel)  to  the  soil  (0-ary)  predicate  P  -FUEL,  and  the  verb  fiul 
{Fuel  fighter  f)  to  die  relational  predicate  P -refuel. 

•  Sonantic  analysis  of  NP  and  VP  adjuncts  as  explicit  roles  or  predicates,  such  as  interpreting  the 
fighter  on  the  carrier  as  “the  fighter  which  is  located  on  the  carrier'’  and  Move  fighter  1  fast  as 
"move  fighter  1  at  higfi  speed.” 

The  TINSEL  semantic  classes  are  organized  into  an  ISA  hierarchy  to  facilitate  encoding  of 
dmnain  type  ctmstraints,  so  subsuming  noun  classes  automatically  become  available  for  use  in  reference: 
b(^  fighters  and  threats  can  be  referred  to  as  aircraft,  agentive  entities  can  be  referenced  as  anyone,  etc. 
Assuming  that  the  users  of  a  tool  that  simulates  a  mUitaty  air  control  system  will  be  comfortable  adhering 
to  a  conshrained  vocabulary  and  regular  syntax,  paraphrase  in  Eucalyptus  is  limited  to  slight  syntactic  vari- 
atitxis  and  some  minor  synonymy  of  verbs  and  prepositions.  Wide  variety  of  usage  is  permitted,  however, 
in  the  referring  expressions  pronouns  and  NPs),  since  discourse  coherence  expresses  itself  so  strongly 
there.  The  reference  resolutitm  module  is  the  next  major  ctnnponent  to  be  discus^. 

Pr^erenees  and  Tran^omuttions 

Two  smaller  modules  not  shown  in  Fig.  2  operate  on  the  TINSEL  ou^mt  before  passing  it  along 
for  reference  resolution  and  further  processing.  The  first  module  weeds  out  semantically  acceptable  but 
inferior  parses  by  applying  heuristic  semantic  preference  rules,  such  as  preferring  parses  as  complete  sen¬ 
tences  rather  than  as  sentence  fragments.  For  example.  Is  a  fighter  moving  to  station  5?,  in  addition  to  the 
expected  sentential  parse,  also  receives  a  less  preferable  analysis  as  die  elliptical  “Is  a  fighter  vthich  is 
moving  to  station  5  [VP]?”  The  second  module  transforms  certain  TINSEL  representations  to  make  diem 
more  suitable  for  conversion  to  logical  form,  for  example  modifying  the  semantic  structure  "Which  fight¬ 
ers  have  a  bearing  of  300?”  into  ‘Tor  which  fighters  does  their  bearing  equal  300?”  The  transformation 
module  is  also  responsible  for  generating  complete  semantic  representadons  for  elliptical  inputs. 

R^erenee  Resolution 

Focus-based  reference  resolution  is  performed  by  the  FOCAL  (FOCus  ALgorithm)  module,  devel- 
(^led  by  ONR  fellow  Gina-Arme  Levow  of  the  Massadiusetts  Institute  of  Technology.  For  each  refetring 
eiqnesaon  in  die  TINSEL  prelogical  form,  FOCAL  graierates  a  data  padret  called  a  Discourse  Entity  con¬ 
taining  the  phrase’s  unique  ID  number,  sonantic  class,  number  (singular/jplural),  die  case  role  it  played  in 
the  utterance,  and  extoisioa  The  extension  is  a  set  of  pointers  to  one  or  mote  symbols,  eadi  rejnesenting  a 
leferenceaUe  entity  in  the  domain.  'The  focusing  algorithm  is  based  largely  on  Refs.  20  and  21  and  uses 
number,  sonantic  dass  [22],  and  recency  to  select  the  most  likely  antecedent  for  eadi  am^dioric  reference. 
The  discourse  oitities  ate  stored  in  data  sttuctiiies  (current  focus,  actor  focus,  alternative  focus  list,  and 
focus  stack)  searched  in  that  order  by  die  focusing  routine  when  trying  to  resolve  possible  amqihora 
(Appendix,  example  4).  As  mentioned  earlier,  definite  references  first  seek  resdution  as  anaidiors,  and  dim 
as  r^rences  to  the  tqipropriate  number  of  objects  from  die  domain  ctMfitext.  Indefinite  extensional  refer- 
mces  {a  fighter,  tme  of  die  threats)  are  assumed  to  represent  “don’t  care”  ^dfications  wdiich,  in  the  con¬ 
text  of  an  imperative,  give  the  system  itself  permission  to  choose  any  mmiber  of  the  extmsion  sd. 

QuantifiettUon 

The  FUNTRAN  Q^TlNctional  TRANslator)  component  transforms  each  TINSEL  case-frame  rep¬ 
resentation  into  an  eiqiresskm  in  first-order  {Medicate  callus  (FOPQ  with  restricted  quantificatitm  (R(^. 
RQ  provides  a  set-ftarmation  operutor  that  allows  comitiex  noun  {diiases  to  be  r^resoited  in  the  logical 
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fonn  as  self-amtained  expiesslims,  unlike  conventional  (e.g.,  Ref.  23)  FOPC  representations  where  the 
predicates  representing  the  NP  are  simfdy  connected  inline  with  those  represoiting  its  parait  clause. 

FUNTRAN  first  establishes  a  scoping  order  over  the  NPs  in  the  expression.  In  cases  of  scoinng 
ambiguity,  FUNTRAN  always  gives  universal  quantifiers  wide  sc(^  and  negation  operates  narrow 
scope.  Hence  Is  a  fighter  vectoring  to  every  threat?  is  always  interpret^  as  asking  whether  each  threat  has 
some  (possildy  different*)  fighter  vectoring  to  it.  This  is  justified  by  the  semantics  of  the  domain,  since 
individual  fighter  actions  in  KOALAS  apply  only  to  singular  goals  (no  fighter  can  simultaneously  vector  to 
more  than  one  threat).  Similarly,  !s  a  fighter  not  vectoring  to  threat  1?  is  assumed  to  ask  whedier  there 
exists  at  least  one  fij^ter  that  is  not  vectoring  to  that  threat,  which  is  the  goierally  preferred  (syntactic) 
scoping. 

The  scoped  NPs  are  then  converted  to  restricted  quantifier  expressions:  universal  and  plural  defi¬ 
nite  NPs  are  mapped  to  FORALL  quantifiers,  indefinite  NPs  to  EXISTS,  and  singular  defitfite  NPs  to 
EXISTS !  (exists-unique).  The  result  is  the  logical  form  of  the  underlying  proposition  of  the  soitence 
(Appendix,  example  5).  The  appropriate  performative  is  then  chosen  based  on  the  syntactic  type  of  the 
utterance:  imperatives  map  to  the  COMMAND  performative,  Wh-questions  to  TELL.  atKfyes/ho  quesfirms  to 
one  of  the  tellif  family  (TELLIF.  tellifany.  tellifall.  and  tellifnone)  depending  on  the 
identity  of  the  outermost  quantifier  (EXISTS ! .  EXISTS.  FORALL,  and  NOT  EXISTS,  respectivdy).^  In 
the  case  of  all  cpteries  but  TELLIF,  the  outermost  quantifier  is  then  converted  to  the  set-fonnation  operator 
SETOF  to  allow  die  performative  to  return  more  informative  answers  than  just  a  flat  “yes"  or  “no.”  The 
performative-matked  quarrtified  expression  (Appendix,  example  6)  is  then  ready  for  evaluation. 

Operator  and  Petformadve  EvaUiatioa 

The  EXISTS,  EXISTS ! ,  and  FORALL  qwrators  are  defined  as  Lisp  macros,  each  with  the  syntax 
(operator  variable  set  test)  .that  expand  into  lambda  erq>ressions  when  evaluated.  EXISTS 
succeeds  if  any  element  of  the  set  passes  the  test,  exists  !  succeeds  if  one  and  only  one  elemem  passes, 
\diile  FORALL  requites  that  every  elonent  pass.  The  SETOF  maoro  has  the  syntax  ( SETOF  variable 
class-or-set  fioptional  test)  and  rehuns  the  subset  of  elemoits  belonging  to  the  class  or  set 
that  pass  the  (optxmal)  test  A  test  can  be  either  another  logical  operator  exi»ession,  or  a  call  to  a  lYansla- 
tkm  Ftmctitm  (such  as  p-MOVE  and  p-HOLD  in  example  6  of  fire  Appendix).  The  evaluatitm  of  Transla- 
tkm  nmctitms  is  discussed  in  the  sectitm  that  follows. 

In  the  case  of  queries,  evaluation  of  the  quantified  expressitm  computes  a  return  value,  dther  a  set 
of  ctmforming  instances  or  a  Bodean  true/false.  This  value  is  then  passed  to  the  enclosing  perfomia- 
tive  operator,  which  rqrorts  the  result  badt  to  die  user  in  some  appropriate  fashion.  The  performatives 
tell,  tellifall,  tellifany  and  tellifnone  c(»npate  the  set  of  (XHiforming  instances  <I>  to  die 
sd  of  exceptimis  <B>  to  produce  a  diort  but  informative  respcmse,  as  shown  in  Talfle  1 .  For  example,  in  die 
evahiadon  of  the  ejqnessitm  in  examffie  6  of  the  Appoidix,  the  set  of  cmifotming  instaiKXS  XI  (all  F14s 
moving  tt>  a  sudon  fighter  1  is  holding)  is  compared  to  the  set  of  exceptitms  N1  -  XI  (where  N1  is  die  set 
of  an  F14s).  If  die  basic  answer  to  the  query  is  “No"  but  some  conforming  instances  do  exist,  the  system 
goes  on  to  list  die  Shorter  of  the  two  sets  (e.g.,  “No,  only  F14  #1  and  F14  #3”).  The  performative  TELLIF 
just  responds  **Yes"  or  “No"  based  cm  the  value  of  its  Boolean  argumoit 


*in  dw  KOALAS  domain,  each  fighter  vectoring  to  a  thrett  miur  be  dififerait,  but  that  is  a  restriction  Uw  pope  quantifier  does  not 
captnie. 

fFtofixmativee  and  (panlifien  for  answering  queatkmB  about  quanthy  {how  many,  how  much,  more  that,  lea  than,  eto.)  are  also 
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Table  1  -  Perfoimative  Behavior 


Perforauitive 

Exaupk 

No 

Instances 

No 

Exceptions 

Fewer 

Instances 

Fewer 

Exceptions 

TELLIFALL 

Are  all  moving? 

No,  none. 

Yes. 

No,  only  <1>. 

No,  not  <E>. 

TELLIFANY 

Are  any  moving? 

No. 

Yes.  all 

Yes,  <I>. 

Yes,  all  but  <E>. 

TELLIFNONE 

Are  none  moving? 

No.  all. 

Only  <I>. 

Allbut<E>. 

TELL 

Which  are  moving? 

None. 

All. 

<I>. 

All  but  <E>. 

TELLIF 

Is  he  moving? 

No. 

Yes. 

AppUcstion-Spedfic  lyanslator 

The  backend  translator  module  provides  the  function  definitions  by  which  NL  predicates  like  P- 
MOVE  and  P-HOLD  query  the  KOALAS  internal  state  or  issue  commands  for  execution  by  the  application. 
The  module  consists  of  two  intercommunicating  sets  of  code,  one  written  in  Liiq)  and  linked  in  with  the 
natural  language  processor,  the  other  writtmi  in  C  and  linked  in  with  KOALAS. 

On  die  Lisp  side.  Eucalyptus  provides  each  TINSEL  veib  class  widi  a  function  definition  called  a 
Translation  Function  (TF).  Whoi  evaluated,  these  expand  into  one  or  more  calls  to  Interface  Functitms 
(IFs),  which  are  primitives  for  interacting  with  die  KOALAS  systmn  and  have  their  code  definitions  on  the 
C  side.  When  a  TF  is  called  in  the  context  of  a  query  or  NP  modifier,  it  translates  into  a  call  to  the  Interface 
Ftincdon  CONFIRM-FACT,  a  {nedicate  (returning  either  TRUE  or  FALSE)  that  consults  KOALAS  to  see 
whether  the  relation  {Medicated  by  the  TF  is  true  of  its  arguments.  When  called  in  the  ctxitext  of  an  impera¬ 
tive  command,  on  die  other  hand,  the  TF  generally  acts  as  a  procedure,  expandiitg  into  a  sequence  of  IF 
calls  diat  cause  KOALAS  state  dianges  to  take  [dace.  For  examide,  in  the  imperative  Move  fighter  1  to  sta¬ 
tion  5,  die  P-MOVE  TF  call  exjrands  into  die  following  sequoice  of  IF  calls: 

(MAKE-MOVE-FIGHTER-POPUP) 

(SET-TYPE-ITEM  F14) 

(SET-ID-ITEM  "1") 

(SET-STATION-ITEM  "5") 

(SET-MESSAGES) 

(SHOW-CURRENT-FRAME) 

As  with  CONFIRM-FACT,  each  of  these  Li^  calls  m^  directly  to  a  corresponding  C  function  linked  into 
KOALAS.  Reflectively,  diey  create  a  “Move  Fighter”  dialogue  window,  set  its  fighter  ^[le  to  the 
KOALAS  ocMistant  F14,  set  its  fighter  ID  ro  1  and  station  ID  to  5,  u{xlate  its  user  {Hvnnpt  message,  and 
finally  make  it  visilde  on  the  screen. 

The  EFs  represent  copies  of  portions  of  already  existing  KOALAS  interface  code  factored  out  for 
oonvenioit  invocation  by  the  Translation  Functions.  In  most  cases,  this  was  done  because  the  original  cotte 
was  intedeaved  with  GUI-idaied  data  structures  such  as  mouse  events  and  needed  to  be  call«l  with  a  dif- 
fierent  parameter  list  In  no  case  was  it  necessaiy  to  create  any  new  data  structures  or  write  truly  novel  code 
fiMT  die  IFIs,  whidi  are  just  copies  and  adfded  portions  of  the  original  source. 


A  lumber  of  KOALAS  command  verbs  have  no  corresponding  data  qaeiy  capability.  For  exam- 
{de,  it  is  possitde  to  order  Have  fighter  1  reUeve  fighter  2,  but  since  the  KOALAS  database  (^y  retains  a 
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record  diat  figjhter  1  is  now  moving  to  fighter  2’s  former  station,  it  cannot  then  answer  the  query  Is  fighter  I 
relieving  fighur  2?  Since  such  a  query  is  semantically  well-fonned  but  pragmatically  outside  the  applica¬ 
tion’s  functional  domain,  the  corresponding  Translation  Function  answers  with  the  message  “I  drni’t 
know,”  in  effect  a  sim[de  kind  of  metaknowledge  (something  the  system  might  be  expected  to  know,  but 
knows  that  it  doesn’t).  A  similar  approach  is  taken  with  questions  that  involve  other  than  simile  present 
tense  (Wm  fighter  1  moving?  Will  he  be  moving?),  which  are  automatically  covered  by  the  NLP  grammar 
and  lexicon  but  lie  outside  the  application’s  ability  to  answer. 

IMPLEMENTATION 

The  NL  processor  (PROTEUS.  TINSEL,  FOCAL,  FUNTRAN,  and  die  semantic  preference  and 
transformadon  modules)  consists  of  about  6860  lines  of  Common  Li^  code.  The  Eucalyptus  translator 
module  contains  about  2940  lines  of  Lisp  code  (Translation  Functions)  and  2480  lines  of  C  code  (Interface 
Functions).  The  KOALAS  application  itself  is  about  161S0  lines  of  C  code.  The  declarative  data  files 
(speech  grammar,  syntactic  grammar  and  lexicon,  %mantic  model,  and  scenario-independent  referent 
model)  total  about  5750  lines  of  text  The  KOALAS  application  requires  3100K  runtime  memory.  Lisp 
6068K,  and  the  Eucalyptus  code  and  data  4684K,  for  a  total  of  13852K,  or  roughly  45%  of  die  available 
memory  on  a  32M  Sun  SPARCstatitni  2. 

In  the  current  version  of  Eucalyptus,  all  the  C  object  code  (KOALAS  itself,  the  speech  decoding 
routines,  and  the  Interface  Functions)  is  linked  into  the  Lisp  process  in  which  the  NL  processor  runs.  The 
Interface  Functions  between  the  two  sets  of  code  are  invok^  as  foreign  function  calls  from  Li^  to  C  rou¬ 
tines  and  vice  versa.  Although  the  simulation  and  NL  processor  run  synchronously,  significant  pauses  in 
the  simulation  are  uncommon  because  the  simulation  display  update  rate  is  once  per  second  and  most 
parses  take  no  longer  than  that.  The  rnily  change  necessary  to  KOALAS  itself  was  the  modification  of  the 
main  simulation  loop  to  poll  for  speech  input  and  send  it  off  to  the  speech  decoding  routines. 

In  another  experimental  versicm,  all  the  C  tdiject  code  was  linked  into  one  process  separate  from 
the  NLP  (Li^)  process.  In  this  version,  the  Interface  Functions  on  each  side  encode  and  send  messages 
(passed  over  su^ard  lA)  channels)  to  the  other  side  for  decoding  and  execution.  This  approach  has  the 
advantage  that  the  NL  processor  and  simulation  nm  asynchronously  (possibly  on  different  machines),  but 
the  message  passing  traffic  in  the  prototype  slowed  processing  times  down  considerably.  For  examine,  a 
query  like  Which  fighters  holding  a  station  that  no  fighters  are  moving  to  are  low  on  fuel?  generates  on  the 
order  of  two  hundred  Interface  Function  calls,  translating  into  double  that  number  of  individual  inteipro- 
cess  messages,  resulting  in  an  unacceptably  loig  processing  time.  More  efficient  message  passing,  1k)w- 
ever,  might  render  this  sq^noach  feasiUe. 

USER  INTERACTION  STUDY 

A  series  of  data  collection  studies  were  conducted  by  Lisa  Achille  at  NRL’s  Human-Computer 
Interaction  Laboratory,  Information  Technology  Division,  after  Eucalyptus  had  already  been  develop^.  In 
the  data  collection  procedure,  a  user  familiar  with  the  KOALAS  tool’s  gnqrhical  interface  issued  verbal 
instructions  and  mouse  pointing  gestures  to  a  second  person,  tire  tool’s  operator,  who  then  executed  tire 
desired  actions  using  the  graftiiical  interface.  The  user’s  verbalizations,  heavily  primed  by  his  familiarity 
with  tire  grrqrhical  interface,  illustrate  a  number  of  issues  relevant  to  the  Eucalyptus  ^roach  and  capabili¬ 
ties  and  ctHifiim  several  of  tire  design  decisions  that  were  made. 

NL  Coverage.  In  addition  to  issuing  simple  NL  equivalents  to  individual  grrqtiucal  inputs  (32),  tire 
user  also  emfrioyed  compounding  (33),  quantification  (34),  anaftiiora  (35).  and  deixis  (36),  all  representing 
NL  features  covered  by  Eucalyptus. 
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(32)  Refuel  5. 

Make  a  threat. 

Put  a  lock  on  4. 

Vector  5  to  track  4  also. 

(33)  Deploy  fighters  9, 8, 7,  and  3. 

Vector  4  u>  12  and  vector  1  to  II. 

Vector  fighters  3  and  9  to  threat  5. 

Recall  11  and  12. 

(34)  Dqtloy  two  more  fighters,  one  in  each  sector. 

Have  all  the  green  fighters  return  to  carrier. 

(35)  Alter  4  and  lock  it. 

Make  a  direat  with  its  radar  return. 

Vector  fighter  1  to  that  threat. 

(36)  Move  1  to  this  sector. 

Make  a  dtreat  here. 

Delete  these  threats  along  in  here. 

Alter  3  to  go  here. 

Anaphora  and  defads.  The  user  etn{doyed  that  for  anaphoric  reference  and  this,  these,  and  here 
ft)r  deixis,  matching  the  approach  taken  in  Eucalyptus. 

Quantification,  ^th  its  strictly  FOPC  quantification,  Eucalyptus  cannot  handle  die  comidex 
quantifier  one  in  each  sector  in  the  sense  intended  in  (33),  i.e.,  dq>loy  each  of  two  fighters  in  a  different 
sector  (implicitly,  sectors  1  and  2). 

Overspedfidty.  In  the  first  sentoice  of  (32),  die  user  consulted  the  Fighter  Status  window  to 
determine  die  ID  numbers  of  the  fighters  still  on  the  carrier,  and  ordered  them  all  dqdoyed  by  naming  them 
individually  in  die  order  they  appeared  in  die  window.  This  reflects  his  prior  knowledge  that  this  is  how  die 
graphical  intofaoe  behaves,  d^oying  (me  individually  idoitified  fighter  at  a  time.  In  Eucalyptus,  die  user 
is  firee  to  say  simidy  Deploy  all  the  fighters  on  the  carrier,  a  direct  exjHession  of  what  s/he  intended  in  die 
first  place. 

Verb  Synonyniy.  In  (37),  the  user  erroneously  issued  a  Vectm*  order  using  the  verb  move  instead 
of  vecior,  but  because  die  destinatkm  is  a  threat,  die  operator  understood  and  executed  the  iraoided  order 
regardless.  The  synonymy  and  type-checking  facilities  of  die  TINSEL  semantic  inteTi»eter  perform  the 
same  function  in  Eucalyptus. 

(37)  Move  fighter  5  to  track  2. 

Ungrammidcality.  Chi  several  occasions,  the  user  ordered  that  a  threat  be  repositioned  using 
ungrammatied  semences  like  (38),  cleaily  [vimed  to  (fo  so  by  die  gnqdiical  interface  vocabulary  and  syn¬ 
tax.  The  incompatildlity  betweoi  this  gn^cal  operator  and  its  NL  verb  structure  was  noted  earlier. 

(38)  Alter  threat  11  to  here. 

Alter  dtreat  6  over  here  to  my  pointer. 

Alter  threat  1  to  the  position  ^fighter  4. 

El^pris.  The  user  often  omitted  the  head  noun  when  referring  to  finders  and  diretus  (Vector  4  to 
/2)t  wl^  Eucalyptus  could  accommodate  with  a  tnmide  addition  to  the  grammar:  The  ordy  occurrences  of 
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vert)  ellipsis  were  in  referaice  to  Advice  Window  button  labels  (39)  and  one  case  of  elided  verb  in  a  fighter 
ctwunand  (40).  In  die  first  case,  it  iqipears  that  the  user  ctHisiders  die  issuing  of  advice  by  the  system  to 
constitute  a  subdialogue,  cmnmanding  focus  (that  is,  a  newly  appearing  piece  of  advice  becomes  the  “cur- 
leid  tqiic'’)  mi  thus  ciqialde  of  taking  elliptical  r^ponses.  In  the  second  case.  Eucalyptus  does  not  cur- 
rendy  handle  ellipticals  containing  prepositional  {diiases  (to  3)  but  otherwise  would  recognize  die  fdirase 
as  a  cmnmand  to  vector  fighter  IS,  since  it  occurred  in  die  cmitext  of  several  prior  Vector  operations. 

(39)  Next. 

Accept. 

(40)  15  to  3. 

Metaknowledge.  At  one  point,  the  user  said  (41)  wishing  to  have  the  screen  redrawn  to  clear  a 
gn^cal  glitch.  While  there  is  no  “refresh”  command,  the  knowledgeable  operator  can  rehesh  the  entire 
screoi  (not  just  a  particular  point)  either  by  recentering  or  zooming  it  In  Eucidyptus,  verbs  like  redraw  or 
r^sh  could  easily  be  added  and  mr^^ied  to  die  t^ropriate  Interface  Function. 

(41)  Refresh  this  point. 

Metalanguage.  The  user  made  a  few  explicit  references  to  the  gn^cal  interface  itself  (42).  This 
is  understandable  since  the  user  knew  that  the  (^rator  was  another  person  viewing  the  scremi,  whereas  in 
Eucalyptus  the  implicit  agent  the  user  is  speaking  to  is  the  KOALAS  system  itself,  which  cannot  “see"  the 
mouse  pointer  icon.  The  system  might  be  expected  to  know  which  finder  icons  are  colored  green,  how¬ 
ever,  and  it  is  quite  simple  in  Eucalyptus  to  just  add  green  as  having  the  same  semantics  as  out  of  weapons, 
v^ch  is  what  that  color  denotes  in  die  di^ay. 

(42)  Alter  threat  6  over  here  to  ms  pointer. 

Have  all  the  green  fighters  return  to  the  carrier. 

RELATED  WORK 

Several  other  systems  have  been  developed  in  recent  years  that  provide  integrated  NL/gnqdiical 
interfaces  widi  deictic  and  other  amqdioric  cqulfilities:  XTRA  [24-27],  Shoptalk  [4, 28],  CHORIS  [29], 
CUBRICX>N  [30-34],  and  the  HITS  user  interfaces  [35].  Of  these,  CHORIS  and  CUBRICON  are  inter¬ 
faces  to  mqi-oriented  ap{dicati(His  and  so  resemUe  ^icalyptus  in  that  regard.  Other  related  systems  of 
interest,  including  earlier  soninal  work  in  NL/gnqihical  integratkm,  can  be  found  in  Refs.  36  through  42. 

XTRA  allows  deictic  referaice  to  the  ctxdents  of  fields  in  a  tax  computatkm  form  interfacing  to  an 
expert  system,  and  [wovides  selectable  granularity  of  pointing  gesture  (exact,  standard,  vague,  and  encir- 
ding). 


Shqptdk  is  an  interfoce  to  a  factory  emulation  that  accqits  deictic  referaice  to  mouse-selected 
icons  represertting  pieces  of  {noduct  and  equipmertt.  As  in  Eucalyptus,  NL  i^uases  can  be  used  to  fill  the 
fields  of  NL-like  dialogue  windows  called  Natural  Language  Form.  Shc^italk  siqiports  use  of  aiuqfiiors  in 
followup  questions  and  also  takes  an  iraiovative  approach  to  the  grqihicd  iderttification  of  discourse  ctm- 
texts. 


CHCWS  is  an  inteUigent  multimedia  interface  that  has  been  adqNed  to  a  geogn^cal  informatitm 
^siem.  Ruses  die  sameNL  i»ocessor  as  Shoptalk  ami  so  supports  deixis  and  grqMcal  discourse  cmdext 
control  miKdi  the  same  way. 
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CUBRICON  is  another  geogra^iical  interface  that,  like  Eucalyptus,  aUows  deictic  reference  to 
icons  and  ^ometric  points  on  a  map;  it  also  accepts  deictic  reference  to  windows  and  table  ouiies,  which 
Eucalyptus  does  not  In  addition.  CUBRICX)N  allows  so-called  “multimodal  NPs,”  where  a  pidc  gesture 
can  take  die  place  of  a  spoken  NP  in  an  utterance  (unlike  deixis,  where  the  two  must  occur  together).  Euca¬ 
lyptus  disallows  multimodal  NPs  because  they  can  result  in  grammatically  ill-fonned  utterances,  which 
nins  ctmtrary  to  Eucalyptus’s  more  highly  NL-oriented  stance. 

The  HITS  (Human  Interface  Tool  Suite)  interface  design  uses  a  distinctive  three-tier  discourse  np- 
resentation  for  multimedia  Kitr  dialogues.  Unlike  Eucalyptus  and  the  other  above-mentioned  systems,  it 
does  not  disamlnguate  graphical  references  using  accompanying  veibal  descriptions,  but  only  accepts  one 
or  the  other.  HITS  provides  a  cognitively  based,  flexible  sc^olding  for  resolving  anaj^ra  and  handling 
incomidetB  inputs. 

Many  of  these  systems  manage  multimedia  inputs  by  insetting  textual  representations  of  grajMcal 
selections  ii^  the  NL  string  being  parsed.  Since  Eucalyptus  only  allows  graphical  inputs  to  accompany 
NL  ii^ts  in  deictic  reference,  it  does  not  require  a  multimedia  parser  but  just  goes  ah^d  and  produces  a 
QpossiUy  amtnguous)  semantic  inteipretation  of  the  veibal  input  and  waits  until  reference  resolution  time 
to  disamlHguaie  it  against  the  (also  possiUy  ambiguous)  graphical  references,  much  as  it  does  in  resolving 
ordinaiy  aniq;^ra. 

CONCLUSIONS 

The  objective  in  Eucalyptus  was  to  demonstrate  the  thesis  that  graphical  and  NL  interfaces  have 
cmnidementaiy  stroigths  and  that  an  integrated  interface  cmnbining  the  two  provides  additional,  natural 
imeiaction  tet^ques.  The  KOALAS  tool’s  original  interface  illustrates  the  GUI’s  strengths  of  tranqnr- 
ency,  lack  of  amUguity,  and  ability  to  easily  denote  spatial  information  (in  this  case,  radar  screen  posi¬ 
tions).  The  NL  interfaw  illustrates  the  strengths  of  flexible  reference,  context-sensitive  abbreviatkai,  and 
freedmn  femn  visil^ty  constraints.  Finally,  deictic  reference  cmnbines  the  naturalness  of  gnqMcal  gesture 
with  that  of  veibal  des^ption  to  provide  a  powerful  imeraction  technique  particulaily  useful  in  msq;>-based 
applications. 

The  KOALAS  tool’s  NL-like  veib-aigument  mechanism  for  issuing  fighter  and  threat  commands 
was  particulaily  suitalfle  for  integratfon  wifli  NL  input  Iixteed,  it  is  likely  the  fool’s  designers  chose  to  use 
that  particular  graf^iical  interaction  tedmitpie  as  a  way  of  Emulating  the  underlying  interactive  task  of 
isso^  (nders  to  fighters,  vriiidi  would  normally  be  done  verbally  were  speech  input  available.  Since  NL  is 
by  its  very  mouie  communicative,  it  is  least  iqipropriate  as  an  alternative  ii^t  modality  to  grtq^cal  inter- 
feces  sriim  direct  manipulation  is  die  most  natural  interactitm  tedmique  (e.g.,  a  paint  program),  becoming 
more  retevaiu  as  the  gn^cal  interactiem  style  beoimes  more  high-level  and  command-orioited,  the  user 
felling  the  ctunpufer  vdiat  to  do  rather  than  having  the  sensation  of  doing  it  him/herself. 

The  KOALAS  tool’s  appmach  to  graidiical  selection  of  radar  screen  objects  (locating  that  object 
dosest  to  tile  mouse  coordinates  that  satisfies  the  semantic  constraints  of  the  expected  aigumoit)  proved  to 
be  highly  adaptabfe  for  use  in  deictic  reference.  The  more  commtm  direct-manipulation  approadi  of  mak- 
the  aiicit^  icems  selectable  objects  by  assod^ng  them  with  mutually  exdusive  mouse-sendtive 
regions  would  not  have  beoi  suitaUe. 

One  strmgth  of  ^aphical  interfaces  is  tiiat  a  new  user  can  explore  the  ^pace  of  pennissiUe  func¬ 
tional  iiqxits  to  an  tqiplicatitm  just  by  traversing  the  imerfece’s  grqtiiical  presentations.  NL  interfaces,  by 
contTMt.  are  typically  qpaque  to  the  user  vtiuit  am  I  permitted  to  say  to  the  Systran,  what  can  we  talk 
ahoitf?  in  Eucalyptus,  tiie  cmnmand  menu  Idiels  suggest  an  acceptaUe  vocabulary  of  imperative  verbs,  tiie 
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dialogue  window  afgument  stiwture  suggests  an  acceptaUe  syntax  of  vert>  arguments,  and  the  nouns  that 
can  be  used  in  those  argumeitts  (fighter,  F14,  threat.  Badger,  etc.)  are  also  to  be  found  in  various  labds, 
disf^ays,  and  pnmqits  throughout  the  gnqMcal  interface.  The  GUI  in  effect  suggests  to  the  user  what  sorts 
of  tliiiigs  can  be  said  to  the  system  and  how  they  can  be  said,  an  ap(»oach  (me  might  call  WYSIWYCS  - 
What  You  See  Is  What  You  Can  Say,  pronounced  “wizzy-wicks”  -  that  ameliorates  the  (^Mcity  i»oUem 
(see  Ref.  40  for  eaiiy  work  in  a  similar  vein). 

Two  commands.  Relieve  and  Alter,  were  found  to  violate  the  WYSIWYCS  i4)ptoach  by  not 
adhering  strictly  to  the  syntax  of  the  corresponding  English  verbs,  and  should  be  redesigned  to  yield  a  fully 
symm^c  integrated  interface.  Similarly,  the  various  widget  types  (chetdt  boxes,  buttons,  and  chcxce 
itmns)  in  the  grqrhical  interface  should  probaUy  have  labels  uniformly  consisting  of  a  verb  and  a  noun 
phrase  (when  transitive),  to  suggest  basic  ^etal  clauses  for  verbal  irqmt  (e.g.,  show  aircraft  trails,  col* 
lect  data,  accept  advice).  In  the  case  of  (dioice  items,  dre  dis{dayed  current  value  would  suggest  the 
optkmal  sectmd  argument  (zoom  screen  to  2,  change  speed  lo  20). 

This  r^roach  also  keeps  the  vocabulary  and  language  model  to  a  size  that  can  be  processed  with 
accqrtably  low  error  rates  by  current  speech  recognition  technology.  The  number  of  referenceable  domain 
(drjects  (typically  100  or  fewer)  and  action  types  (about  80)  in  the  KOALAS  tool  is  also  well  matched  to 
the  technology.  The  introduction  of  flexible  reference,  however,  greatly  increases  the  input  qrace  and  the 
chances  of  ^reech  recognition  errors  through  the  (xmfiision  of  similar  sounding  pronouns  and  d^rminers 
(e.g..  a,  the,  duu,  diem,  this,  these,  those).  If  the  q)eech  recognition  grammar  could  have  accommodated 
relative  clauses,  recognition  errors  would  have  increased  even  furdier,  but  perhaps  not  to  an  uncomfbrtaUe 
level.  Providing  for  several  restricted  types  of  elliptical  ir^t  also  pushed  the  speech  recognitiem  require¬ 
ments  consideratfly,  but  the  overall  system  is  still  sufSdendy  accurate  for  comfortable  use. 

The  ellipsis  and  anafdiora  handling  ciqMdMlity  of  the  discourse  component  increases  the  naturalness 
and  ease  of  use  of  the  NL  interface  considetaUy.  Since  the  graphical  interface  generally  (nohiUts  more 
dun  one  subdialogue  frcmi  occurring  at  a  time,  the  corre^xmding  NL  interface  can  get  by  for  die  most  part 
with  a  sim(de  discourse  model  employing  (uily  a  single  (xmtext  space.  In  graphical  interfaces  v^re  focus 
can  ddft  firom  (me  window  to  ant^r,  however,  n^thanisms  for  directing  NL  input  to  particular  subdia- 
l(^ues  (windows)  will  be  retiuired. 

Eucalyptus’s  requirement  that  all  verbal  exchanges  with  the  system  resemble  an  inteipersomd  NL 
diatogue  is  too  coiKtraining:  the  machine  is  a  fool,  not  a  perstm,  and  users  may  want  to  be  quite  tose  with 
it  wbenevo'  posable  and  only  resort  to  foil  NL  when  diey  feel  the  need  to  exfdain  or  (daiify  stxnediing. 
Th^  will  thus  use  a  much  wider  range  of  telegraphic,  foagmentary,  and  iU-formed  uttnances  with  the 
machine  dian  commtmly  occurs  in  intopersonal  discewrse  [43].  This  suggests  that  a  (xmilrinatitm  of  ordi¬ 
nary  s|)ee(h  input  (e.g.,  qiealdng  menu  itons)  and  Eucalyptus-style  sptdten  NL  is  desiralde  in  muldmodal 
interfaces,  which  may  require  cutting  back  (m  the  amount  of  allowable  ellipdcal  iiqut 

Finally,  Eucalyptus’s  modular  (tesign  proved  its  worth  when  the  interface  was  successfully  ported 
to  an  XView-based  version  of  the  KOALAS  tool  in  less  than  (me  working  day.  The  key  to  this  success  was 
die  bderface  Ftmctkm  module,  rqneseiding  KOALAS  code  needed  by  Eucalyptus  but  factored  out  from 
the  qmdfics  of  the  GUI’s  grrqhics  routines. 
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Appendix 

DATA  PROCESSING  EXAMPLES 


1.  SSI  Fhcmetic  Decoder  ou^ 


•r*  all  tha  f  fourtaana  moving  to  a  station  fightar  1  is  holding 

2.  PROTEUS  imennediate  syntactic  lepresoitation 

(reqoest  present  prog  hove 

(ALL  N1  F14  PLORAL) 

(TO  (SOME  M2  STATION  SIMGOLAR 
(PRESENT  PROG  HOLD 

(MOLL-DET  N3  FIGHTER  SIMGOLAR  (IDMOM  1)) 

VAR)))) 

3.  TINSEL  semantic  ii)teq[)fetati(Ni 

(REQUEST  PRESENT  PROG  VI  (: CLASS  P-MOVE) 

('.PATIENT  (ALL  Ml  (: CLASS  P-F14  PLORAL))) 

(:TO-LOC  (SOME  M2  (: CLASS  P-STATION)  SINGULAR 
(PRESENT  PROG  V2  (: CLASS  P-HOU» 

(sPATIEMT  (NOLL-l»T  N3  (:CLASS  P-FIGHTER) 
SINGULAR  (:ID  1))) 

(:AT-LOC  M2))))) 


4.  FOCAL  aUemate  focus  list 

((HI  SINGULAR  P-FIGHTER  : PATIENT  (FRIENDLY-1  FRIEMDLY-2  ...  FRlEHDLY-9) ) 
(N2  SINGULAR  P-STATION  :TO-LOC  (STATION-2)) 

(N3  SINGULAR  P-F14  :PATIENT  (FRIENOLY-1) ) ) 

5.  FUNTRAN  quantified  ejqxession 

(FORALL  XI  (SETOF  N1  P-F14) 

(EXISTS  X2  (SETOF  N2  P-STATICNI 

(EXISTS!  X3  (SETOr  N3  P-FIGHTER  (:ID  1)) 
(P-HOLD  :PATIENT  X3  :AT-LOC  N2))) 
(P-MOVE  tPATIENT  XI  :TO-LOC  X2))) 

6.  Quantified  expiesskm  afim-poformative  insertion 

(TBLLIFALL 

(SETOF  XI  (SETCar  N1  P-F14) 

(EXISTS  X2  (SETOF  M2  P-STATION 

(EXISTS!  X3  (^TOr  N3  P-FKNiTER  (:ID  1)) 
(P-HOLD  :PATIEMT  X3  :AT-LOC  M2))) 
(P-MOVE  :PATIENT  XI  :TO-LOC  X2))) 
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